Functional Reactive Programming in the Netflix API - QCon London 2013

The Netflix API receives over two billion requests a day from more than 800 different devices ranging from gaming consoles like the PS3, XBox and Wii to set-top boxes, TVs and mobile devices such as Android and iOS.

As the unique demands of different devices have diverged it became clear that the API needed to allow optimizing client-server interaction. We achieve this by enabling the creation of customized service endpoints that reduce network chatter, leverage server-side processing power and decentralize the development of each endpoint so as to encourage and empower rapid innovation.

This presentation will describe how the Netflix API achieves these goals using functional reactive programming (modeled after Rx) in a polyglot Java stack. Highly concurrent service endpoints are implemented without blocking, synchronization or thread-safety concerns by using declarative functional reactive composition. Parallelism and resiliency are enabled by the underlying Hystrix fault tolerance isolation layer that fronts the dozens of distributed systems within the Netflix SOA.

Presented March 6th at QCon London
http://qconlondon.com/london-2013/speaker/Ben+Christensen

Transcript

1. Functional Reactive Programming in the Netflix API Ben Christensen Software Engineer – API Platform at Netflix @benjchristensen http://www.linkedin.com/in/benjchristensen http://techblog.netflix.com/ QCon London – March 6 2013 1
2. 2 Netflix is a subscription service for movies and TV shows for $7.99USD/month (about the same converted price in each countries local currency).
3. More than 33 million Subscribers in more than 50 Countries and Territories 3 Netflix has over 33 million video streaming customers in 50 countries and territories across North & South America, United Kingdom, Ireland and the Nordics.
4. Netflix accounts for 33% of Peak Downstream Internet Traffic in North America Netflix subscribers are watching more than 1 billion hours a month 4 Sandvine report available with free account at http://www.sandvine.com/news/global_broadband_trends.asp
5. API traffic has grown from ~20 million/day in 2010 to >2 billion/day 0 500 1000 1500 2000 2010 2011 2012 Today millions of API requests per day 5
6. Discovery Streaming 6 Streaming devices talk to 2 major edge services: the first is the Netflix API that provides functionality related to discovering and browsing content while the second handles the playback of video streams.
7. Netflix API Streaming 7 This presentation focuses on architectural choices made for the “Discovery” portion of traffic that the Netflix API handles.
8. 8 The Netflix API powers the “Discovery” user experience on the 800+ devices up until a user hits the play button at which point the “Streaming” edge service takes over.
9. Open API Netflix Devices API Request Volume by Audience 9 Greater than 99.9% of traffic to the Netflix API is focused on serving the discovery UIs of Netflix streaming devices. This means it is primarily an internal API used by Netflix development teams.
10. Netflix API Dependency A Dependency D Dependency G Dependency J Dependency M Dependency P Dependency B Dependency E Dependency H Dependency K Dependency N Dependency Q Dependency C Dependency F Dependency I Dependency L Dependency O Dependency R 10 The Netflix API serves all streaming devices and acts as the broker between backend Netflix systems and the user interfaces running on the 800+ devices that support Netflix streaming. 2+ billion incoming calls per day are received which in turn fans out to several billion outgoing calls (averaging a ratio of 1:6) to dozens of underlying subsystems.
11. /ps3/home Dependency F 10 Threads Dependency G 10 Threads Dependency H 10 Threads Dependency I 5 Threads Dependency J 8 Threads Dependency A 10 Threads Dependency B 8 Threads Dependency C 10 Threads Dependency D 15 Threads Dependency E 5 Threads Dependency K 15 Threads Dependency L 4 Threads Dependency M 5 Threads Dependency N 10 Threads Dependency O 10 Threads Dependency P 10 Threads Dependency Q 8 Threads Dependency R 10 Threads Dependency S 8 Threads Dependency T 10 Threads /android/home /tv/home Functional Reactive Dynamic Endpoints Asynchronous Java API 11
12. /ps3/home Dependency F 10 Threads Dependency G 10 Threads Dependency H 10 Threads Dependency I 5 Threads Dependency J 8 Threads Dependency A 10 Threads Dependency B 8 Threads Dependency C 10 Threads Dependency D 15 Threads Dependency E 5 Threads Dependency K 15 Threads Dependency L 4 Threads Dependency M 5 Threads Dependency N 10 Threads Dependency O 10 Threads Dependency P 10 Threads Dependency Q 8 Threads Dependency R 10 Threads Dependency S 8 Threads Dependency T 10 Threads /android/home /tv/home Functional Reactive Dynamic Endpoints Asynchronous Java API Hystrix fault-isolation layer 12 Backend communication to underlying services is isolated using Hystrix (https://github.com/Netflix/Hystrix)
13. /ps3/home Dependency F 10 Threads Dependency G 10 Threads Dependency H 10 Threads Dependency I 5 Threads Dependency J 8 Threads Dependency A 10 Threads Dependency B 8 Threads Dependency C 10 Threads Dependency D 15 Threads Dependency E 5 Threads Dependency K 15 Threads Dependency L 4 Threads Dependency M 5 Threads Dependency N 10 Threads Dependency O 10 Threads Dependency P 10 Threads Dependency Q 8 Threads Dependency R 10 Threads Dependency S 8 Threads Dependency T 10 Threads /android/home /tv/home Functional Reactive Dynamic Endpoints Asynchronous Java API 13 This presentation is going to focus on why the Netflix API team chose the functional reactive programming model (Rx in particular), how it is used and what benefits have been achieved. Other aspects of the Netflix API architecture can be found at http://techblog.netflix.com/search/label/api and https://speakerdeck.com/benjchristensen/.
14. RxJava “a library for composing asynchronous and event-based programs using observable sequences for the Java VM” A Java port of Rx (Reactive Extensions) https://rx.codeplex.com (.Net and Javascript by Microsoft) 14
15. Do we really need another way of “managing” concurrency? 15
16. Discovery of Rx began with a re-architecture ... 16 More information about the re-architecture can be found at http://techblog.netflix.com/2013/01/optimizing-netflix-api.html
17. ... that collapsed network traffic into coarse API calls ... Nested, conditional, parallel execution 17 Within a single request we now must achieve at least the same level of concurrency as previously achieved by the parallel network requests and preferably better as we can leverage the power of server hardware, lower latency network communication and eliminate redundant calls performed per incoming request.
18. ... and we wanted to allow anybody to create endpoints, not just the “API Team” 18 User interface client teams now build and deploy their own webservice endpoints on top of the API Platform instead of the “API Team” being the only ones who create endpoints.
19. 19 We wanted to retain flexibility to use whatever JVM language we wanted as well as cater to the differing skills and backgrounds of engineers on different teams. Groovy was the first alternate language we deployed in production on top of Java.
20. Concurrency without each engineer reading and re-reading this -> (awesome book ... everybody isn’t going to - or should have to - read it though, that’s the point) 20
21. Owner of API should retain control of concurrency behavior. 21
22. public Data getData(); What if the implementation needs to change from synchronous to asynchronous? How should the client execute that method without blocking? spawn a thread? Owner of API should retain control of concurrency behavior. 22
23. public void getData(Callback<T> c); public Future<T> getData(); public Future<List<Future<T>>> getData(); What about ... ? 23
24. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted() 24 Observable/Observer is the asynchronous dual to the synchronous Iterable/Iterator. More information about the duality of Iterable and Observable can be found at http://csl.stanford.edu/~christos/pldi2010.fit/meijer.duality.pdf and http://codebetter.com/ matthewpodwysocki/2009/11/03/introduction-to-the-reactive-framework-part-ii/
25. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted()  //  Iterable<String>    //  that  contains  75  Strings  getDataFromLocalMemory()    .skip(10)    .take(5)    .map({  s  -­‐>        return  s  +  "_transformed"})    .forEach(          {  println  "next  =>  "  +  it})  //  Observable<String>    //  that  emits  75  Strings  getDataFromNetwork()    .skip(10)    .take(5)    .map({  s  -­‐>        return  s  +  "_transformed"})    .subscribe(          {  println  "onNext  =>  "  +  it}) 25 The same way higher-order functions can be applied to an Iterable they can be applied to an Observable.
26. Iterable pull Observable push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted()  //  Iterable<String>    //  that  contains  75  Strings  getDataFromLocalMemory()    .skip(10)    .take(5)    .map({  s  -­‐>        return  s  +  "_transformed"})    .forEach(          {  println  "onNext  =>  "  +  it})  //  Observable<String>    //  that  emits  75  Strings  getDataFromNetwork()    .skip(10)    .take(5)    .map({  s  -­‐>        return  s  +  "_transformed"})    .subscribe(          {  println  "onNext  =>  "  +  it}) 26
27. Instead of blocking APIs ... class  VideoService  {      def  VideoList  getPersonalizedListOfMovies(userId);      def  VideoBookmark  getBookmark(userId,  videoId);      def  VideoRating  getRating(userId,  videoId);      def  VideoMetadata  getMetadata(videoId); } class  VideoService  {      def  Observable<VideoList>  getPersonalizedListOfMovies(userId);      def  Observable<VideoBookmark>  getBookmark(userId,  videoId);      def  Observable<VideoRating>  getRating(userId,  videoId);      def  Observable<VideoMetadata>  getMetadata(videoId); } ... create Observable APIs: 27 With Rx blocking APIs could be converted into Observable APIs and accomplish our architecture goals including abstracting away the control and implementation of concurrency and asynchronous execution.
28. 28 For example, an Observable API could just use the calling thread to synchronously execute and respond.
29. 29 Or it could use a thread-pool to do the work asynchronously and callback with that thread.
30. 30 Or it could use multiple threads, each thread calling back via onNext(T) when the value is ready.
31. 31 Or it could use an actor pattern instead of a thread-pool.
32. 32 Or NIO with an event-loop.
33. 33 Or a thread-pool/actor that does the work but then performs the callback via an event-loop so the thread-pool/actor is tuned for IO and event-loop for CPU. All of these different implementation choices are possible without changing the signature of the method and without the calling code changing their behavior or how they interact with or compose responses.
34. Observable.toObservable("one",  "two",  "three")          .take(2)          .subscribe((arg)  -­‐>  {                    System.out.println(arg);          }); Java8 Observable.toObservable("one",  "two",  "three")    .take(2)    .subscribe((arg:  String)  =>  {            println(arg)    }) Scala (-­‐>      (Observable/toObservable  ["one"  "two"  "three"])    (.take  2)      (.subscribe  (fn  [arg]  (println  arg)))) Clojure    Observable.toObservable("one",  "two",  "three")        .take(2)          .subscribe({arg  -­‐>  println(arg)}) Groovy    Observable.toObservable("one",  "two",  "three")        .take(2)          .subscribe(lambda  {  |arg|  puts  arg  }) JRuby 34 Simple examples showing RxJava code in the 5 languages supported as of RxJava 0.5 (https://github.com/Netflix/RxJava/tree/master/language-adaptors). Java8 works with rxjava-core and does not need a language-adaptor. It also works with Java 6/7 but without lambdas/closures the code is more verbose.
35.        Observable.create({  observer  -­‐>            try  {                  observer.onNext(new  Video(id))                observer.onCompleted();            }  catch(Exception  e)  {                observer.onError(e);            }        }) 35 Let’s look at how to create an Observable and what its contract is. An Observable receives an Observer and calls onNext 1 or more times and terminates by either calling onError or onCompleted once. More information is available at https://github.com/Netflix/RxJava/wiki/Observable
36.        def  Observable<VideoRating>  getRating(userId,  videoId)  {                //  fetch  the  VideoRating  for  this  user  asynchronously                return  Observable.create({  observer  -­‐>                        executor.execute(new  Runnable()  {                                def  void  run()  {                                    try  {                                          VideoRating  rating  =  ...  do  network  call  ...                                        observer.onNext(rating)                                        observer.onCompleted();                                      }  catch(Exception  e)  {                                        observer.onError(e);                                      }                                      }                        })                })        } Asynchronous Observable with Single Value 36 Example Observable implementation that executes asynchronously on a thread-pool and emits a single value.
37. Synchronous Observable with Multiple Values        def  Observable<Video>  getVideos()  {                return  Observable.create({  observer  -­‐>                      try  {                              for(v  in  videos)  {                                observer.onNext(v)                          }                          observer.onCompleted();                      }  catch(Exception  e)  {                          observer.onError(e);                      }                })        } Caution: This is eager and will always emit all values regardless of subsequent operators such as take(10) 37 Example Observable implementation that executes synchronously and emits multiple values. Note that the for-loop as implemented here will always complete so should not have any IO in it and be of limited length otherwise it should be done with a lazy iterator implementation or performed asynchronously so it can be unsubscribed from.
38. Asynchronous Observable with Multiple Values  def  Observable<Video>  getVideos()  {        return  Observable.create({  observer  -­‐>              executor.execute(new  Runnable()  {                    def  void  run()  {                        try  {                                for(id  in  videoIds)  {                                  Video  v  =  ...  do  network  call  ...                                  observer.onNext(v)                              }                              observer.onCompleted();                          }  catch(Exception  e)  {                              observer.onError(e);                          }                      }              })        })  } 38 Example Observable implementation that executes asynchronously on a thread-pool and emits multiple values. Note that for brevity this code does not handle the subscription so will not unsubscribe even if asked. See the ‘getListOfLists'  method  in the following for an implementation with unsubscribe handled: https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-groovy/src/ examples/groovy/rx/lang/groovy/examples/VideoExample.groovy#L125
39. Observable<SomeData> a = getDataA(); Observable<SomeData> b = getDataB(); Observable<SomeData> c = getDataC(); Observable.merge(a, b, c) .subscribe( { element -> println("data: " + element)}, { exception -> println("error occurred: " + exception.getMessage())} ) Combining via Merge 39 How to combine Observable sequences of the same type using the ‘merge’ operator.
40. Observable<SomeData> a = getDataA(); Observable<String> b = getDataB(); Observable<MoreData> c = getDataC(); Observable.zip(a, b, c, {x, y, z -> [x, y, z]}) .subscribe( { triple -> println("a: " + triple[0] + " b: " + triple[1] + " c: " + triple[2])}, { exception -> println("error occurred: " + exception.getMessage())} ) Combining via Zip 40 How to combine Observable sequences of different types using the ‘zip’ operator.
41. Observable<SomeData> a = getDataA(); Observable<String> b = getDataB(); Observable<MoreData> c = getDataC(); Observable.zip(a, b, c, {x, y, z -> [x, y, z]}) .subscribe( { triple -> println("a: " + triple[0] + " b: " + triple[1] + " c: " + triple[2])}, { exception -> println("error occurred: " + exception.getMessage())} ) Error Handling 41 If an error occurs then the ‘onError’ handler passed into the ‘subscribe’ will be invoked.
42. Observable<SomeData> a = getDataA(); Observable<String> b = getDataB(); Observable<MoreData> c = getDataC() .onErrorResumeNext(getFallbackForDataC()); Observable.zip(a, b, c, {x, y, z -> [x, y, z]}) .subscribe( { triple -> println("a: " + triple[0] + " b: " + triple[1] + " c: " + triple[2])}, { exception -> println("error occurred: " + exception.getMessage())} ) Error Handling 42 Errors can be handled on each sequence similar to a try/catch rather than it causing the entire combined sequence to fail. Various ‘onError*’ operators can be found in the Javadoc: http:// netflix.github.com/RxJava/javadoc/rx/Observable.html
43. def Observable getVideos(userId) { return VideoService.getVideos(userId) } Asynchronous request that returns Observable<Video> 43 Now for a more involved example that demonstrates some of the power of Rx to handle nested asynchronous composition.
44. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) } Reactive operator on the Observable that takes the first 10 Video objects then unsubscribes. 44
45. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .map({ Video video -> // transform video object }) } The ‘map’ operator allows transforming the input value into a different output. 45
46.        Observable<R>  b  =  Observable<T>.map({  T  t  -­‐>              R  r  =  ...  transform  t  ...            return  r;        }) 46 The ‘map’ operators allows transforming from type T to type R.
47. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... }) } 47
48. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... }) } We change to ‘mapMany’ which is like merge(map()) since we will return an Observable<T> instead of T. 48
49.  Observable<R>  b  =  Observable<T>.mapMany({  T  t  -­‐>          Observable<R>  r  =  ...  transform  t  ...        return  r;  }) 49 The ‘mapMany’ operator allows transforming from type T to type Observable<R>. If ‘map’ were being used this would result in an Observable<Observable<R>> which is rarely what is wanted, so ‘mapMany’ flattens this via ‘merge’ back into Observable<R>. This is generally used instead of ‘map’ anytime nested work is being done that involves fetching and returning other Observables.
50. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... }) } Nested asynchronous calls that return more Observables. 50
51. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... }) } Observable<VideoMetadata> Observable<VideoBookmark> Observable<VideoRating> 51 3 separate types are being fetched asynchronously.
52. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... }) } Each Observable transforms its data using ‘map’ 52
53. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> // for each video we want to fetch metadata def m = video.getMetadata() .map({ Map<String, String> md -> // transform to the data and format we want return [title: md.get("title"), length: md.get("duration")] }) // and its rating and bookmark def b ... def r ... // compose these together }) } 53
54. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> def m ... def b ... def r ... // compose these together }) } 54
55. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> def m ... def b ... def r ... // compose these together return Observable.zip(m, b, r, { metadata, bookmark, rating -> // now transform to complete dictionary // of data we want for each Video return [id: video.videoId] << metadata << bookmark << rating }) }) } 55
56. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> def m ... def b ... def r ... // compose these together return Observable.zip(m, b, r, { metadata, bookmark, rating -> // now transform to complete dictionary // of data we want for each Video return [id: video.videoId] << metadata << bookmark << rating }) }) } The ‘zip’ operator combines the 3 asynchronous Observables into 1 56
57.        Observable.zip(a,  b,  {  a,  b,  -­‐>              ...  operate  on  values  from  both  a  &  b  ...            return  [a,  b];  //  i.e.  return  tuple        }) 57
58. def Observable<Map> getVideos(userId) { return VideoService.getVideos(userId) // we only want the first 10 of each list .take(10) .mapMany({ Video video -> def m ... def b ... def r ... // compose these together return Observable.zip(m, b, r, { metadata, bookmark, rating -> // now transform to complete dictionary // of data we want for each Video return [id: video.videoId] << metadata << bookmark << rating }) }) } return a single Map (dictionary) of transformed and combined data from 4 asynchronous calls 58
59. 59 Now we will walk through the same flow but using a marble diagram instead of code to show what happened and how the sequences and functions interacted.
60. Observable<Video> emits n videos to onNext() 60
61. Takes first 10 then unsubscribes from origin. Returns Observable<Video> that emits 10 Videos. 61
62. For each of the 10 Video objects it transforms via ‘mapMany’ function that does nested async calls. 62
63. For each Video ‘v’ it calls getMetadata() which returns Observable<VideoMetadata> These nested async requests return Observables that emit 1 value. 63
64. The Observable<VideoMetadata> is transformed via a ‘map’ function to return a Map of key/values. 64
65. Same for Observable<VideoBookmark> and Observable<VideoRating> 65 Each of the .map() calls emits the same type (represented as an orange circle) since we want to combine them later into a single dictionary (Map).
66. The 3 ‘mapped’ Observables are combined with a ‘zip’ function that emits a Map with all data. 66
67. The full sequence emits Observable<Map> that emits a Map for each of 10 Videos. 67
68. Client code treats all interactions with the API as asynchronous The API implementation chooses whether something is blocking or non-blocking and what resources it uses. 68
69. Example of latency reduction achieved by increasing number of threads used by Observables. 69 A performance issue was discovered in a relatively large endpoint doing a lot of work. We added a property to allow dynamically changing how many threads were thrown at it so we could adjust it in production at runtime. This graph shows how latency reduced ~25% by increasing the number of threads working on that particular code. Since the endpoint was implemented with Rx and was functionally ‘pure’ without external mutation it was safe and no calling code needed to change to allow this to happen.
70. + Observable<User>  u  =  new  GetUserCommand(id).observe(); Observable<Geo>  g  =  new  GetGeoCommand(request).observe(); Observable.zip(u,  g,  {user,  geo  -­‐>                  return  [username:  user.getUsername(),                                  currentLocation:  geo.getCounty()]       }) RxJava coming to Hystrix https://github.com/Netflix/Hystrix 70 Hystrix will support RxJava. See https://github.com/Netflix/Hystrix/issues/123.
71. <dependency  org="com.netflix.rxjava"  name="rxjava-­‐core"  rev="x.y.z"  /> <dependency  org="com.netflix.rxjava"  name="rxjava-­‐groovy"  rev="x.y.z"  /> <dependency  org="com.netflix.rxjava"  name="rxjava-­‐clojure"  rev="x.y.z"  /> <dependency  org="com.netflix.rxjava"  name="rxjava-­‐scala"  rev="x.y.z"  /> <dependency  org="com.netflix.rxjava"  name="rxjava-­‐jruby"  rev="x.y.z"  /> <dependency>        <groupId>com.netflix.rxjava</groupId>        <artifactId>rxjava-­‐core</artifactId>        <version>x.y.z</version> </dependency> ... or for a different language ... To get started ... 71 See https://github.com/Netflix/RxJava/wiki/Getting-Started for more information.
72. Functional Reactive in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html Optimizing the Netflix API http://techblog.netflix.com/2013/01/optimizing-netflix-api.html RxJava https://github.com/Netflix/RxJava @RxJava Ben Christensen @benjchristensen http://www.linkedin.com/in/benjchristensen Netflix is Hiring http://jobs.netflix.com 72