Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scala Use Cases at Hatena

mechairoi
September 06, 2014

Scala Use Cases at Hatena

はてなでベータ版を提供中のサーバ管理サービスであるMackerelは、サーバサイドの開発に Scala, Play2 を採用しています。10年以上Perlを利用してきたはてなが、なぜ新たな開発言語としてScalaを選択したのか、言語の変化がプロダクトや開発フローにどのような影響あたえたのか、現在のMackerelの運用・開発手法などを紹介します。

mechairoi

September 06, 2014
Tweet

Other Decks in Technology

Transcript

  1. Takaya Tsujikawa (@mechairoi) blog.chairoi.me mechairoi ! - 2009 Hatena Intern

    (Perl) - 2011 Joined Hatena (Perl) - 2013~ Mackerel team lead engineer (Scala)
  2. Table of Contents • Web development with Perl & problems

    • Why Scala? • Web development with Scala • Pros & Cons
  3. • Internet Company 2001~ • Hatena Bookmark • Hatena Blog

    • Flipnote Hatena etc. • Almost products by Perl
  4. – http://www.paulgraham.com/hp.html “We need a language that lets us scribble

    and smudge and smear, not a language where you have to sit with a teacup of types balanced on your knee and make polite conversation with a strict old aunt of a compiler”
  5. Why Perl? (200x) • CPAN • Rapid prototyping • Code

    density ⛳️ • Duck typing • Perl Community (YAPC etc)
  6. Resistance to collapse • Coding rule • Code review •

    Test / Continuous integration • Testing is require but not sufficient • Static code analysis IUUQTXXXqJDLSDPNQIPUPT!/
  7. Background • 2013/11 Start • 1 engineer + Outsource •

    B2B • Low dependency to existing system.(ex. Account system) • Owned service • Less restriction Chance of introduction of new technology
  8. Requirements • Modern type system • Lots of libraries •

    Case studies of web development • Member’s Skill
  9. Why Scala (1/4) • Modern type system • Static typing

    • Algebraic data type (case class, sealed trait) • Type inference • Ad hoc polymorphism (implicit conversion) • B2B Safe refactoring and improvement
  10. Why Scala (4/4) • Member’s skills • some members had

    been studied/was interested in Scala • Product must not depend on individual skills
  11. Other languages • Java8 • Not available yet • Haskell

    • Hard to assemble team • Go/Ruby • No modern type system
  12. • Fullstack WAF • Routing • Type-safe template engine •

    play.api.{data.Form, libs.Json} • Pretty error messages • play-auth/stackable-controller
  13. • Type safe template engine (Twirl) • Our designer writes

    Scala expression • Hard for designer • Rewriting Angularjs template • 2.2.1 → 2.2.2 → 2.3.1
  14. Design policy • MVC • Classes • object models.Users #create

    • Side effects • case class models.User (email: String, … ) • Immutable data container • No side effects, no variable.
  15. Design policy • Less variable, less mutable collection, less partial

    function ( Option#get ) • User defined types (Tagged Type) • UserId / HostId instead of Int • UserName instead of String Increase guarantee by type
  16. Design policy • Mapping primitive type and domain types by

    type classes )551CPEZ ! 63* ! %BUBTUPSF "QQMJDBUJPO .PEFM .BQQJOHCZ UZQFDMBTT 4USJOH +40/  KBWBMBOH42- DBTFDMBTTFT 6TFS )PTU
  17. Routing GET /hosts/:id controllers.Hosts.retrieve(id: HostId) implicit def hostIdFormat: Formatter[HostId] =

    new Formatter[HostId] { def bind(key: String, data: Map[String, String]) : Either[Seq[FormError], HostId] = ??? ! def unbind(key: String, value: HostId) = ??? } <a href=“@(routes.Hosts.retrieve(hostId)”>…</a>
  18. Modules • Modularize for compile speed ( In progress..) $PSF5ZQFT

    $PSF .PEFMT $PSF$POUSPMMFST $PSF 7JFXT "MFSU5ZQFT "MFSU .PEFMT "MFSU$POUSPMMFST "MFSU 7JFXT 3PPU 3FWFSTF 3PVUFS
  19. • Schema code generation • Type-safe queries and raw queries

    • All Queries in our product are type-safe • Extends for “FOR UPDATE OF t1” • 1.0.1 → 2.0.0 → 2.0.3 ( → 2.1.0)
  20. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap
  21. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for {! (log, logLater) <- Tables.alertLogs(orgId) leftJoin! Tables.alertLogs(orgId) on { case (log, logLater) =>! log.createdAt < logLater.createdAt &&! log.alertId === logLater.alertId! }! if log.alertId inSetBind entities.map(_.id)! if logLater.createdAt.?.isNull! } yield (log.alertId, log.status)).list.toMap
  22. Build complex queries easily and safely val statusById: Map[AlertId, MonitorStatus]

    = (for { (log, logLater) <- Tables.alertLogs(orgId) leftJoin Tables.alertLogs(orgId) on { case (log, logLater) => log.createdAt < logLater.createdAt && log.alertId === logLater.alertId } if log.alertId inSetBind entities.map(_.id) if logLater.createdAt.?.isNull } yield (log.alertId, log.status)).list.toMap
  23. Not human readable select x2.x3, x2.x4 from (select x5."updated_at" as

    x6, x5."status" as x4, x5."alert_id" as x3, x5."created_at" as x7, x5."operator_ id" as x8, x5."id" as x9, x5."trigger" as x10 from "alert_logs" x5, "alerts" x11 where (x11."id" = x5."alert_id") and (x11."org_id" = ?)) x2 left outer join (select x12."updated_at" as x13, x12."status" as x14, x12."alert_id" as x15, x12."created_at" as x16, x12."operator_id" as x17, x12."id" as x18, x12."trigger" as x19 from "alert_logs" x12, "alerts" x20 where (x20."id" = x12."alert_id") and (x20."org_id" = ?)) x21 on (x2.x7 < x21.x16) and (x2.x3 = x21.x15) where (x2.x3 in (?)) and (x21.x16 is null)
  24. Not human readable select x2.x3, x2.x4 from (select x5.x6 as

    x7, x5.x8 as x9, x5.x10 as x11, x5.x12 as x13, x5.x14 as x15, x5.x16 as x17, x5.x18 as x3, x5.x19 as x20, x5.x21 as x22, x5.x23 as x24, x5.x25 as x26, x5.x27 as x28, x5.x29 as x30, x5.x31 as x32, x5.x33 as x34, x5.x35 as x36, x37.x38 as x39, x37.x40 as x4, x37.x41 as x42, x37.x43 as x44, x37.x45 as x46, x37.x47 as x48, x37.x49 as x50, x37.x51 as x52 from (select x53."updated_at" as x6, x53."image_url" as x8, x53."hosts_count_limit" as x10, x53."webhook_url" as x12, x53."enabled_experimental_features" as x14, x53."created_at" as x16, x53."id" as x18, x53."owner_team_id" as x19, x53."name" as x21, x53."gravatar_email" as x23, x53."is_staff" as x25, x53."notification_activates_at" as x27, x53."plan_id" as x29, x53."plan_expired_at" as x31, x53."author_id" as x33, x53."token" as x35 from "orgs" x53) x5 left outer join (select x54."updated_at" as x38, x54."agv_count" as x40, x54."max_count" as x41, x54."total" as x43, x54."created_at" as x45, x54."base_datetime" as x47, x54."id" as x49, x54."org_id" as x51 from "host_metrics_counts" x54) x37 on x5.x18 = x37.x51) x2 left outer join (select x55."updated_at" as x56, x55."agv_count" as x57, x55."max_count" as x58, x55."total" as x59, x55."created_at" as x60, x55."base_datetime" as x61, x55."id" as x62, x55."org_id" as x63 from "host_metrics_counts" x55) x64 on (x2.x3 = x64.x63) and (x2.x48 < x64.x61) where ((x2.x3 in (?)) and (x64.x61 is null)) and (x2.x48 > {ts '2014-09-02 17:27:18.039'}) • Where does code generate this query?
  25. In Perl case • No type-safe • Publisher class in

    SQL Comment SELECT * FROM user WHERE user_id = ?; -- Hatena::User
  26. In Perl case _人人人人人人人人人_ >�Merge without conflicts�< ‾Y^Y^Y^Y^Y^Y^Y^Y‾ Can't locate

    object method "retrieve_by_id" via package "Entry" at error.pl line 21. Entry->retrieve_by_id(10); Entry->retrieve(id => 10); Entry->retrieve_by_id(20); 3FGBDUPSJOH "EEFEDBMMT
  27. Scrum • Sprint = Milestone • Burn down chart •

    radekstepan/github-burndown-chart
  28. git-pr-release • Auto generate pull-request • master ← develop •

    Check list / issues • Everyone can deploy • motemen/git-pr-release
  29. Deploy • Restarting JVM is slow → 502 • Capistrano

    • Custom deploy strategy • scp jar files from Jenkins • Rolling deploy • daemontools https://www.flickr.com/photos/gsfc/9807812154/in/photolist-fWFARy-dBeVvk-96iJmd-atSZgv-hFZzw8-aoA2hH-j2aiC2-mrQV5-kwW6fT-9MP2Ju-edaJM7-M3WWJ-kmUfnB-dxxRza- Cjb4u-4YSS4-8JpmiE-4yjdpB-bbibXF-btu7MA-bGoWje-kTaXqi-jJpjHn-5RV54-4rMctW-bGoWnD-ecFBPp-9Yf9KD-9XsGPA-LVDt-aRcVjM-5xgexk-9S6d4F-4y4qNv-eZwRu-osQq9g-ebMkug- ed9XJo-qyM6W-aUyKZH-4cxiEU-bGoWsk-btu7F1-776kcJ-9zuC1x-j2bH1R-bypWd7-e6BdUN-bGoWeX-btu7A7
  30. Pros • Peace of Mind • Refactoring/Upgrade is easy •

    Can focus to architecture/design and logic in review • Functional programming • Collection libraries
  31. Cons • Require Java/JVM literacy • Rich language functions •

    Hard to learn • De-facto standard Job queue? • Mocking is difficult in tests