Scala Exchange 2014: Netflix + Scala

Find out the architecture Netflix (producers of such fine entertainment as "Lillyhammer" & "Arrested Development") is using for their next generation device metadata APIs – specifically, the tech stack & development+deployment processes.

We will explore demonstrate real world use cases which make Scala stand out as a choice programming language for for building clean, beautiful, RESTful APIs with tools such as Swagger+Scalatra. We will also explore Scala testing frameworks, and how they plug into our larger CI/CD infrastructure, with additional discussions of Netflix OSS components.

Transcript

1. Netflix + Scala Brendan McAdams (@rit) bmcadams@netflix.com
2. What My Team Does a.k.a. "What do you use Scala for?"
3. APIs! » RESTful services comprising business & domain logic » UI built on top of REST using AngularJS » Scalatra with Swagger
4. Collecting Valuable Knowledge » Analyze information about devices running Netflix before public release » Display Capabilities: 3D? 720p? 1080p? UltraHD (4K)? » Remote Control: Dedicated Netflix button? » Lots of other metadata of value to Netflix
5. Collecting Valuable Knowledge » Used widely internally » Certification: run accurate tests » UI: Optimum look, feel, and behavior » Support: debug customer issues
6. Why We Love Scala » Concise yet expressive language » Hybridisation of OO and FP provides flexibility » Great libraries and toolchains » Solid Java interop » Stays crunchy in milk
7. The Right Tool(s) for the Job » Plays well with Netflix' Platform & JVM ecosystem » Reuse existing Java & Groovy code » Lots of powerful Java libraries in the Netflix toolbelt, easily accessible from Scala
8. The Right Tool(s) for the Job » Production containers to inject healthchecks, management hooks, etc... » Scalatra deploys into existing infrastructure as standard J2EE app » Existing Gradle build system & Jenkins CI well suited for JVM languages
9. The Right Tool(s) for the Job » A lot of Netflix' core tools are open source » A few: » Simian Army: Chaos Monkey helps ensure resilient systems » Asgard: AWS Service Management » Hysterix: Latency & Fault Tolerance helpers » http://netflix.github.io/
12. A Little Swagger In Our Step » Swagger makes API Development a lot easier for us » Since another group of developers interacts with our service » They have interactive, detailed documentation at their fingertips
13. A Little Swagger In Our Step » Swagger is built around self documenting code » Documentation is always up to date. » Swagger itself is mostly a JSON based "Specification" » Scalatra module for describing API, generates JSON » swagger-ui JSON Frontend explores API
14. Swagger at a Glance
15. "GET" get("/", operation(getTestAccounts)) { withAuthenticatedUser { authUser: AuthenticatedUser => val testAccounts = TestAccountLogManager.findByUser( authUser ) Ok( ApiResponse(testAccounts) ) } }
16. "GET" def getTestAccounts = authApiOperation[TestAccountResponse]( "getTestAccounts", """Returns a list of the test accounts created by the current user with this tool.""")
17. "GET" def getTestAccounts = authApiOperation[TestAccountResponse]( "getTestAccounts", """Returns a list of the test accounts created by the current user with this tool.""") » Swagger automatically documents the API from the authApiOperation method... » The base Swagger method is apiOperation, which we've extended internally to support our auth layer.
21. Creating a Test Account post("/", operation(createTestAccount)) { withAuthenticatedUser { user: AuthenticatedUser => withValidJsonBody[TestAccountRequest] { testAccountReq => createAcctForUser(user, testAccountReq ) } } }
22. Creating a Test Account def createTestAccount = authApiOperation[String]("createTestAccount", """Creates a test account, returning JSON result from the API""") .parameter( bodyParam[TestAccountRequest]("") .description( """The request includes the new Netflix test account's country of origin, email address, password and (test) user's first name. """) )
23. "POST" def createTestAccount = authApiOperation[String]("createTestAccount", "Creates a test account, returning JSON result from the API") .parameter( bodyParam[TestAccountRequest]("").description( """The request includes the new Netflix test account's country of origin, email address, password and (test) user's first name. """) ) » In the case of more complex methods, we can include parameter information » e.g. bodyParam, pathParam, queryParam, headerParam
25. Swagger documents our bodyParam
26. Hitting Model Schema shows a JSON schema of the Parameter
27. Software Deployments
28. Traditional Software Deployment » The "traditional" J2EE deployment model » WAR files deployed to Tomcat/Jetty/My Lovecraftian J2EE Horror
29. Traditional Software Deployment » Software changes? Just redeploy to the same old server » Rolling back to old versions isn't always straight forward » Chasing dependency conflicts is never fun
30. Traditional Software Deployment » Reproducibility is low: If the server changes, hard to solve problems » Did the new build break? » Did someone muck with the software on the server w/o telling anyone? » Was the server invaded by Gibbering Horrors from Beyond Spacetime?
31. Immutable Deployments A Better Way » Build server images as immutable AMIs » Never modified once deployed » Need to roll back? Pick an image of the last known working version...
32. Immutable Deployments » Building an image requires a valid Jenkins build » Only Jenkins can merge to 'master'
33. Immutable Deployments » In this setup, good testing is key to success » We have tools to manage the deployment of server images
36. Immutable Deployments » Asgard (open source) helps manage Amazon services, including Auto Scaling Groups » Servers grouped into clusters ensures that each deployment group (e.g. Dev, Staging, Production) are of one cohesive version » Last working version available as a hot spare / backup
39. Smaller is Better » Build lots of small services » Move quickly, move independently » APIs are powerful in this model » API can feed UI, Data Services, etc without a monolithic dev cycle
40. Good Testing Matters » This should go without saying, but good tests matter quite a lot » Especially in an environment where you're building immutable images... » And relying upon CI to determine the validity of a build
41. Good Testing Matters » We use and love ScalaTest <scalatest.org> » Specs2 <specs2.org> is pretty awesome as well
42. Scalatra Testing » Scalatra ships with some nice test helpers for ScalaTest, Specs2, and Scalacheck » How would we test that TestAccountService with ScalaTest?
43. Scalatra Testing class TestAccountServiceSpec extends BaseScalatraFlatSuite with Logging { addServlet(new TestAccountService, "/*") "Test Account Service" should "verify if email addresses use the Netflix domain" in { val body = """{ "country":"US", "password":"pass", "fname":"ScalaX" "email":"scalax-demo@somedomain.com", }""" postJsonWithToken("/", body)(TestUserTokens.brendanToken) { status should equal(BadRequest().status.code) log.debug("Returned, status = %s, body = %s", status, body) } } }
44. Testing Your API » Your API Framework of choice should provide good tools for testing it » If it doesn't, consider finding a better framework » Scalatra (the framework we're using) provides great bindings to Scalatest » This contributes greatly to feeling comfortable that what you've built works as documented
45. Write Lots of Tests » Positive Tests » Negative Tests » Service Level Tests (HTTP Calls) » Helper Level Tests (backend objects sans HTTP) » Parser Tests » Regression Tests » Voight-Kampff Tests
46. "Never Trust A Test You Haven't Seen Fail." - Rob Fletcher ( @rfletcherEW ) ... who says he nicked it from @tddmonkey (Colin Vipurs)
47. Errors! Errors! Errors!
48. ERRORS » Meaningless error messages lead to long days of slaving over error logs » Nothing sucks more than spending an hour digging through logs... » ... only to find it was something you could have communicated with an error message
49. ERRORS » Meaningless error messages lead to long days of slaving over error logs » Nothing sucks more than spending an hour digging through logs... » ... only to find it was something you could have communicated with an error message
50. Good Practices for Errors » If you can validate an entire request object, do so » Return every error you know about immediately, and clearly » If you can pass multiple errors up your stack and return them, do so... » JSON Supports Arrays, after all
51. Good Practices for Errors » Provide useful data, not a generic 500 Internal Server Error » If you have an exception, extract a message from it » However: Exceptions are not flow control » Some useful tools out there in the Scala world for Error handling & passing...
52. We are Not Using the 'Zed' Word
53. We are Not Using the 'Zed' Word
54. scalaz is kind of awesome... » We've been playing with scalaz a bit lately » There's a lot in there, and a lot to absorb » The bits I've found most useful so far? » Validation » Disjunction ( \/ )
55. problems to solve » Providing clearer errors & validating input was a problem » API Server is part of a larger application: erroring is hard » 500s & generic exceptions, complicate frontend devs debugging
56. Helping developers help themselves » An Error Occurred » API received bad/invalid data? » Database failed? » What if multiple errors occurred? » How do we communicate this effectively?
57. » Validation + NonEmptyList is great for "These are all the things that went wrong" » Validation is Applicative: you can accrue the results of many Validations together » Using a NonEmptyList for Failure means the applicative adds each error » Operators for doing cool things » Implicits to convert things
58. » \/ (Disjunction), like Scala's Either, has a "Left" and a "Right" value » Unlike Scala's Either, \/ is Monadic: You can iterate over it » This is "right biased": like Option it unwraps if it is "Right" » Operators and Implicits for converting from things like Option
59. Come to my talk tomorrow: 'A Skeptic's Look at scalaz' "Gateway Drugs”': In Room 2 @ 1600!
60. Questions?