Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OSCON 2014 - API Ecosystem with Scala, Scalatra...

OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix

In this talk I’d like to introduce the Scala-based API stack at Partner Innovation Group at Netflix. After seeing a massive growth in the business model and the device ecosystem, we needed a system that could scale and be flexible at the same time. Scala provided the answer and we started with a basic set of APIs which, since then, has evolved towards complex but flexible business flows. Supporting metadata for over hundreds of brands and thousands of devices, the API development has followed a well thought-out, test-driven approach, git-flow, and what most API developers dread – documentation. I will talk about the architecture of the RESTful APIs, and the development + deployment process. We use Netflix-OSS components heavily in the architecture and cloud deployment, so I will cover them as well.

Swagger is what we used for type-safe documentation, which is really easy to use and integrate. I will briefly talk about customizations we’ve done to Swagger in order to make it far more usable at Netflix.

Throughout this effort there were lessons to be learnt, and plenty of best practices and recommendations for anyone starting out to build RESTful APIs, regardless of the platform or stack of choice. It’d be a great opportunity for me to walk through the architecture, and talk about the various components, technologies, and practices that are seeing increasing adoption in the modern, API driven landscape.

Manish Pandit

July 23, 2014
Tweet

More Decks by Manish Pandit

Other Decks in Technology

Transcript

  1. Power the Consumer Electronics Partner Portal Enable Certification of Netflix

    Ready Devices Source of truth for all Device Data at Netflix Correlate Streaming Quality Metrics Our APIs
  2. Model Firmware Screen Resolution Subtitle Support 3D DRM Remote Control

    Netflix SDK 4K HD …   Everything matters..
  3. Background Netflix OSS Components Development Deployment/Delivery Open Floor Architecture HTTP

     Layer,  and  Manager  Layer   Cassandra EVCache   Crowd/SSO RDS Astyanax Netflix OSS Cloud Components
  4. HTTP Layer = Protocol Wrapper Map HTTP methods to Manager

    functions Transform JSON <=> Case Classes
  5. Background Netflix OSS Components Development Deployment/Delivery Open Floor Open Source

    Infrastructure Components Libraries Tools Frameworks PaaS with AWS IaaS
  6. /** * Map results from a java.sql.ResultSet to a list

    of objects of another type * @param rs The result set * @param f Function that takes the result set and returns an instance of T * @tparam A The type * @return Sequence of instances of type A */ def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = { val array = ArrayBuffer[A]() while (rs.next) { array += f(rs) } array.toSeq }
  7. /** * Function to return a cached value, or fetch

    it per the parameter f * @param key The key to look up in cache, or set against this key if not found * @param f The function that fetches the instance from some persistent store * @tparam A The type * @return None if not found in cache and persistent store, else Some[A] */ def withCache[A](key: String)(f: => Option[A]): Option[A] = { EVCacheGateway.get[A](key) match { case None => val data = f if (!data.isEmpty) EVCacheGateway.set(key, data.get) data case Some(x) => Some(x) } }
  8. /** * Get a user given an ID * @param

    id The User ID * @return None if user is not found, else the user */ def getById(id: Long): Option[User] = { withCache[User](s"User$id") { val query = "select * from users where user_id = ?" DBHelper.queryWith(query, extract, id).headOption } }
  9. /** * Get all the chipsets ordered by name. */

    get("/providers", operation(getChipsetProviders)) { Ok( ChipsetManager.getProviders(0,20) ) }
  10. it should "return the chipset provider partner orgs" in {

    val data = ChipsetManager.getProviders(0,20) data.records.isEmpty should be (false) data.records.size should be (20) data.total should be >= 21l }
  11. ScalatraSpec Background Netflix OSS Components Development Deployment/Delivery Open Floor Traits

    to take ScalaTest to the next level Helpers for JSON parsing Plenty of wrappers (body, headers..)
  12. addServlet(new ChipsetService, "/*") it should "get a list of chipset

    providers" in { getWithToken("/providers")(TestUserTokens.testToken) { status should equal (200) body should include (""""start":0,"end":19,"count":20""") } }
  13. get("/providers", operation(getAllChipsetProviders)) { //code } case class PartnerList(data: Seq[PartnerOrg], start:

    Int, end: Int, count: Long, total: Long) def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all chipset providers") .parameter(queryParam[Option[Int]]("start").description("Starting record count, defaults to 0")) .parameter(queryParam[Option[Int]]("count").description("The number of records requested, defaults to 20"))
  14. Deployment Background Netflix OSS Components Development Deployment/Delivery Open Floor Push

    to dev Jenkins runs dev build, tests, merges to master Jenkins runs master build, makes a .deb Aminator bakes an AMI from the .deb Asgard deploys the AMI in staging cloud
  15. lgml-mpandit:nrd-portal-api mpandit$ git status On branch dev Your branch is

    up-to-date with 'origin/dev'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scala modified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scala modified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scala modified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala no changes added to commit (use "git add" and/or "git commit -a") Check in
  16. [info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386

    [success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PM Build step 'Build using sbt' changed build result to SUCCESS Pushing HEAD to branch master of origin repository Returning node parameter for ssh-dynaslave-1291624c Triggering a new build of PPD-NRD-PORTAL-API-MASTER #1172 Notifying upstream projects of job completion Finished: SUCCESS Dev Build
  17. Build Artifacts: nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB [fingerprint] Changes: Added an endpoint

    to return a list of chipset provider partner orgs. NFLX-5514 (detail) Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923 originally caused by: Started by an SCM change Revision: 880e1873fef63d278b3180b49af7434e234dee40 origin/master Master Build
  18. Summary Background Netflix OSS Components Development Deployment/Delivery Open Floor Bridge

    the gap between dev and deploy No such thing as too many tests Automate everything Document your APIs Best solutions are implementation agnostic