Scala Use Cases at Hatena

はてなでベータ版を提供中のサーバ管理サービスであるMackerelは、サーバサイドの開発に Scala, Play2 を採用しています。10年以上Perlを利用してきたはてなが、なぜ新たな開発言語としてScalaを選択したのか、言語の変化がプロダクトや開発フローにどのような影響あたえたのか、現在のMackerelの運用・開発手法などを紹介します。

1. Scala Use Cases at Hatena גࣜձࣾ͸ͯͳ Takaya Tsujikawa https://www.flickr.com/photos/30081184@N02/3769383676
2. Takaya Tsujikawa (@mechairoi) blog.chairoi.me mechairoi ! - 2009 Hatena Intern (Perl) - 2011 Joined Hatena (Perl) - 2013~ Mackerel team lead engineer (Scala)
3. Table of Contents • Web development with Perl & problems • Why Scala? • Web development with Scala • Pros & Cons
5. • Internet Company 2001~ • Hatena Bookmark • Hatena Blog • Flipnote Hatena etc. • Almost products by Perl
6. Why Perl?
7. – http://www.paulgraham.com/hp.html “We need a language that lets us scribble and smudge and smear, not a language where you have to sit with a teacup of types balanced on your knee and make polite conversation with a strict old aunt of a compiler”
8. Why Perl? (200x) • CPAN • Rapid prototyping • Code density ⛳️ • Duck typing • Perl Community (YAPC etc)
9. a decade later … https://www.flickr.com/photos/josh/11029669/in/photolist-YwJH-dZrHB3-dZro3Y-dZkJSF-dZrn6b-dZkTkH-dZm2se-dZkN5R-dZrFnG-dZm1Lc-aEwYeQ-8cKHdX-aGrYEa-593eo9-eUa2aU- eZXyHe-f1cTP9-dZkQzP-dZrJps-dZkYtg-dZm3gp-dZkAwV-dZkyLc-dZrhAo-dZrjiC-dZrgmN-dZrifq-dZri1Y-dZkS2a-dZkVoM-eUar2C-dZrhLu-a3hhH4-dZruES-eTXXbc-dZkUhB-dZkCJF- dZrmTY-dZrCXS-dZkEYz-dZkZsz-dZkZJZ-dZrrTb-dZkGH2-dZrkZ5-dZrp1N-dZrpNG-dZrHYQ-dZrq5C-dZrsA1/
10. Web applications become more complex https://www.flickr.com/photos/adrian_s/8271860/in/photolist-JoWh-2J1ULn-5MBjL1-7Jb7po-PBm6n-4YxhWQ-ftWYrC-56q8A8-byH7Vo-g6K6Gv-aiH2Kq-oqCruv-wJ9B-6qtmTj- bzvmcN-8SLbUW-4EUk2k-7WsSCV-6WR5CX-bQkpxT-EN7qe-5i3Tcf-aUrm1H-8ZRy8r-aBUFd2-6jBeaS-8N6ZoM-ig1HkK-4Ut5PG-8t8Q6D-5KjrEv-4cTcqP-3feujT-7Bpbqh- atG3JH-22M7W-8cwtgh-5jQDxs-EN7nt-cc87U3-6Zmv7D-4q13tN-9MqM4o-dK4pWN-8agRiA-GK1af-crBgSU-7NVEST-9Lg7jB-6KRFX Business Library Update Legacy Code New features Mobile
11. Resistance to collapse • Coding rule • Code review • Test / Continuous integration • Testing is require but not sufficient • Static code analysis IUUQTXXXqJDLSDPNQIPUPT!/
12. ʊਓਓਓਓਓਓਓਓਓਓਓਓਓʊ ʼɹSudden Runtime Errorɹʻ ʉY^Y^Y^Y^Y^Y^Y^Y^Y^Y^Y^Yʉ
13. We need static typing https://www.flickr.com/photos/hetgacom/10420400556/in/photostream/
14. Why xxScala?
15. New Project!
16. A New Kind of Application Performance Management
17. Push architecture host1 host2 host3 Mackerel Agent
18. Graphs by roles cpu usage of proxies loadavg5 of apps
19. Monitoring • Mail • Webhook
20. Background • 2013/11 Start • 1 engineer + Outsource • B2B • Low dependency to existing system.(ex. Account system) • Owned service • Less restriction Chance of introduction of new technology
21. 
    
    
    
    
    
    
    
    
    Go
    
    
    
    
    
    
    
    
    
    
    
    Graphite
    
    
    
    
    
    
    PostgreSQL
    
    
    
    
    
    
    
    
22. 
    
    
    
    
    
    
    
    
    Go
    
    
    
    
    
    
    
    
    
    
    
    Graphite
    
    
    
    
    
    
    PostgreSQL
    
    
    
    
    
    
    
    
23. Requirements • Modern type system • Lots of libraries • Case studies of web development • Member’s Skill
24. Why Scala (1/4) • Modern type system • Static typing • Algebraic data type (case class, sealed trait) • Type inference • Ad hoc polymorphism (implicit conversion) • B2B Safe refactoring and improvement
25. Why Scala (2/4) • Lots of libraries
26. Why Scala (3/4) • Case studies of web development
27. Why Scala (4/4) • Member’s skills • some members had been studied/was interested in Scala • Product must not depend on individual skills
28. Other languages • Java8 • Not available yet • Haskell • Hard to assemble team • Go/Ruby • No modern type system
29. Development with Scala
30. • Fullstack WAF • Routing • Type-safe template engine • play.api.{data.Form, libs.Json} • Pretty error messages • play-auth/stackable-controller
31. • Type safe template engine (Twirl) • Our designer writes Scala expression • Hard for designer • Rewriting Angularjs template • 2.2.1 → 2.2.2 → 2.3.1
32. Design policy • MVC • Classes • object models.Users #create • Side effects • case class models.User (email: String, … ) • Immutable data container • No side effects, no variable.
33. In Perl case ! Side effects are used everywhere
34. Design policy • Less variable, less mutable collection, less partial function ( Option#get ) • User defined types (Tagged Type) • UserId / HostId instead of Int • UserName instead of String Increase guarantee by type
35. In Perl case ! No static typing Just try hard
36. Design policy • Mapping primitive type and domain types by type classes )551
    CPEZ
    ! 63*
    ! %BUB
    TUPSF "QQMJDBUJPO
    
    .PEFM .BQQJOH
    CZ
    UZQF
    DMBTT 4USJOH
    +40/
    KBWBMBOH42- DBTF
    DMBTTFT
    
    6TFS
    )PTU
37. Routing GET /hosts/:id controllers.Hosts.retrieve(id: HostId) implicit def hostIdFormat: Formatter[HostId] = new Formatter[HostId] { def bind(key: String, data: Map[String, String]) : Either[Seq[FormError], HostId] = ??? ! def unbind(key: String, value: HostId) = ??? } <a href=“@(routes.Hosts.retrieve(hostId)”>…</a>
38. Modules • Modularize for compile speed ( In progress..) $PSF
    5ZQFT $PSF
    .PEFMT $PSF
    $POUSPMMFST $PSF
    7JFXT "MFSU
    5ZQFT "MFSU
    .PEFMT "MFSU
    $POUSPMMFST "MFSU
    7JFXT 3PPU 3FWFSTF
    3PVUFS
39. • Schema code generation • Type-safe queries and raw queries • All Queries in our product are type-safe • Extends for “FOR UPDATE OF t1” • 1.0.1 → 2.0.0 → 2.0.3 ( → 2.1.0)
40. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus] = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap
41. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus] = (for {! (log, logLater) <- Tables.alertLogs(orgId) leftJoin! Tables.alertLogs(orgId) on { case (log, logLater) =>! log.createdAt < logLater.createdAt &&! log.alertId === logLater.alertId! }! if log.alertId inSetBind entities.map(_.id)! if logLater.createdAt.?.isNull! } yield (log.alertId, log.status)).list.toMap
42. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus] = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap
43. Not human readable select x2.x3, x2.x4 from (select x5."updated_at" as x6, x5."status" as x4, x5."alert_id" as x3, x5."created_at" as x7, x5."operator_ id" as x8, x5."id" as x9, x5."trigger" as x10 from "alert_logs" x5, "alerts" x11 where (x11."id" = x5."alert_id") and (x11."org_id" = ?)) x2 left outer join (select x12."updated_at" as x13, x12."status" as x14, x12."alert_id" as x15, x12."created_at" as x16, x12."operator_id" as x17, x12."id" as x18, x12."trigger" as x19 from "alert_logs" x12, "alerts" x20 where (x20."id" = x12."alert_id") and (x20."org_id" = ?)) x21 on (x2.x7 < x21.x16) and (x2.x3 = x21.x15) where (x2.x3 in (?)) and (x21.x16 is null)
44. Not human readable select x2.x3, x2.x4 from (select x5.x6 as x7, x5.x8 as x9, x5.x10 as x11, x5.x12 as x13, x5.x14 as x15, x5.x16 as x17, x5.x18 as x3, x5.x19 as x20, x5.x21 as x22, x5.x23 as x24, x5.x25 as x26, x5.x27 as x28, x5.x29 as x30, x5.x31 as x32, x5.x33 as x34, x5.x35 as x36, x37.x38 as x39, x37.x40 as x4, x37.x41 as x42, x37.x43 as x44, x37.x45 as x46, x37.x47 as x48, x37.x49 as x50, x37.x51 as x52 from (select x53."updated_at" as x6, x53."image_url" as x8, x53."hosts_count_limit" as x10, x53."webhook_url" as x12, x53."enabled_experimental_features" as x14, x53."created_at" as x16, x53."id" as x18, x53."owner_team_id" as x19, x53."name" as x21, x53."gravatar_email" as x23, x53."is_staff" as x25, x53."notification_activates_at" as x27, x53."plan_id" as x29, x53."plan_expired_at" as x31, x53."author_id" as x33, x53."token" as x35 from "orgs" x53) x5 left outer join (select x54."updated_at" as x38, x54."agv_count" as x40, x54."max_count" as x41, x54."total" as x43, x54."created_at" as x45, x54."base_datetime" as x47, x54."id" as x49, x54."org_id" as x51 from "host_metrics_counts" x54) x37 on x5.x18 = x37.x51) x2 left outer join (select x55."updated_at" as x56, x55."agv_count" as x57, x55."max_count" as x58, x55."total" as x59, x55."created_at" as x60, x55."base_datetime" as x61, x55."id" as x62, x55."org_id" as x63 from "host_metrics_counts" x55) x64 on (x2.x3 = x64.x63) and (x2.x48 < x64.x61) where ((x2.x3 in (?)) and (x64.x61 is null)) and (x2.x48 > {ts '2014-09-02 17:27:18.039'}) • Where does code generate this query?
45. In Perl case • No type-safe • Publisher class in SQL Comment SELECT * FROM user WHERE user_id = ?; -- Hatena::User
46. Development flow
47. IDE AND / OR
48. Git Branches operation NBTUFS EFWFMPQ GFBUVSF GFBUVSF 3FMFBTF
49. Git Branches operation NBTUFS EFWFMPQ GFBUVSF GFBUVSF 3FMFBTF No accident on merge
50. In Perl case ＿人人人人人人人人人＿ ＞�Merge without conflicts�＜ ‾Y^Y^Y^Y^Y^Y^Y^Y‾ Can't locate object method "retrieve_by_id" via package "Entry" at error.pl line 21. Entry->retrieve_by_id(10); Entry->retrieve(id => 10); Entry->retrieve_by_id(20); 3FGBDUPSJOH "EEFE
    DBMMT
51. Task management • Github issues/pull requests • Tag • Milestone
52. Scrum • Sprint = Milestone • Burn down chart • radekstepan/github-burndown-chart
53. Jenkins/CI • Test & Packaging parallel • 1h → 20min • Git integration
54. git-pr-release • Auto generate pull-request • master ← develop • Check list / issues • Everyone can deploy • motemen/git-pr-release
55. Deploy • Restarting JVM is slow → 502 • Capistrano • Custom deploy strategy • scp jar files from Jenkins • Rolling deploy • daemontools https://www.flickr.com/photos/gsfc/9807812154/in/photolist-fWFARy-dBeVvk-96iJmd-atSZgv-hFZzw8-aoA2hH-j2aiC2-mrQV5-kwW6fT-9MP2Ju-edaJM7-M3WWJ-kmUfnB-dxxRza- Cjb4u-4YSS4-8JpmiE-4yjdpB-bbibXF-btu7MA-bGoWje-kTaXqi-jJpjHn-5RV54-4rMctW-bGoWnD-ecFBPp-9Yf9KD-9XsGPA-LVDt-aRcVjM-5xgexk-9S6d4F-4y4qNv-eZwRu-osQq9g-ebMkug- ed9XJo-qyM6W-aUyKZH-4cxiEU-bGoWsk-btu7F1-776kcJ-9zuC1x-j2bH1R-bypWd7-e6BdUN-bGoWeX-btu7A7
56. In Perl case • Capistrano • git clone • Server::Starter • Graceful restart
57. Scala Pros & Cons
58. Pros https://www.flickr.com/photos/quinndombrowski/5200218267/in/photolist-6k8go4-a3y271-8VwtHD-skiqN-nN34SF-bveV1T-cjNRK9-oddpGc-6SXagE-fea96C-6MLPeT-a7pJhj-wDorR- a55adA-8uYbhg-4HP2Qm-dhuorr-aYXd4F-eLGkd-e4eFKc-9ugzpc-6MZ9UU-bWXfCV-9GH9mD-59UEA7-aG95zM-3fJmjC-cHPX65-8aBfdr-j7GYKE-e6ViAm-bAhx1p-feQRPa-92JZzp-kLtFHa- XWd9L-9zbeGs-9DdEqR-8maR73-nBgwNS-8WFsTq-7wSNi8-MvvjP-5DMyrq-8Kowmj-54rxHi-BqUvC-fLbCYs-7F9di2-nR8Kt4
59. Pros • Peace of Mind • Refactoring/Upgrade is easy • Can focus to architecture/design and logic in review • Functional programming • Collection libraries
60. Cons • Slow compilation • modularize • high spec machine
61. Cons • Require Java/JVM literacy • Rich language functions • Hard to learn • De-facto standard Job queue? • Mocking is difficult in tests
62. Conclusion ! Cannot go back to Perl
63. Any Question?
64. We are hiring