Share
Embed
Reactive Programming with Rx at QConSF 2014
Reactive programming in the form of RxJS and RxJava has been adopted at Netflix over the past couple years for both client-side and server-side programming. This talk will summarize why the Rx programming model was chosen and demonstrate how it is applied to a variety of use cases including web services, interprocess communication and stream processing. Along the way topics like flow control, backpressure, concurrency, and event loops will be discussed and how they play into building responsive, resilient, scalable applications.
Video: http://www.infoq.com/presentations/rx-service-architecture
More information can be found at www.reactivex.io or www.github.com/ReactiveX/RxJava
Ben Christensen
November 04, 2014
Transcript
175 slides
-
Ben Christensen Developer – Edge Engineering at Netflix @benjchristensen http://techblog.netflix.com/
QConSF - November 2014 Reactive Programming with Rx -
Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T)
onError(Exception) onCompleted() -
Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T)
onError(Exception) onCompleted() // Iterable<String> or Stream<String> // that contains 75 Strings getDataFromLocalMemory() .skip(10) .limit(5) .map(s -‐> s + "_transformed") .forEach(t -‐> System.out.println("onNext => " + t)) // Observable<String> // that emits 75 Strings getDataFromNetwork() .skip(10) .take(5) .map(s -‐> s + "_transformed") .forEach(t -‐> System.out.println("onNext => " + t)) -
Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T)
onError(Exception) onCompleted() // Observable<String> // that emits 75 Strings getDataFromNetwork() .skip(10) .take(5) .map(s -‐> s + "_transformed") .forEach(t -‐> System.out.println("onNext => " + t)) // Iterable<String> or Stream<String> // that contains 75 Strings getDataFromLocalMemory() .skip(10) .limit(5) .map(s -‐> s + "_transformed") .forEach(t -‐> System.out.println("onNext => " + t)) -
Single Multiple Sync T getData() Iterable<T> getData() Stream<T> getData() Async
Future<T> getData() Observable<T> getData() -
// add onError and onComplete listeners Observable.just(“Hello”, “World!”) .subscribe(System.out::println, Throwable::printStackTrace,
() -> System.out.println("Done")); -
// expand to show full classes Observable.create(new OnSubscribe<String>() { @Override
public void call(Subscriber<? super String> subscriber) { subscriber.onNext("Hello World!"); subscriber.onCompleted(); } }).subscribe(new Subscriber<String>() { @Override public void onCompleted() { System.out.println("Done"); } @Override public void onError(Throwable e) { e.printStackTrace(); } @Override public void onNext(String t) { System.out.println(t); } }); -
// add error propagation Observable.create(subscriber -> { try { subscriber.onNext("Hello
World!"); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println); -
// add error propagation Observable.create(subscriber -> { try { subscriber.onNext(throwException());
subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println); -
// add error propagation Observable.create(subscriber -> { try { subscriber.onNext("Hello
World!"); subscriber.onNext(throwException()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println); -
// add concurrency (manually) Observable.create(subscriber -> { new Thread(() ->
{ try { subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).start(); }).subscribe(System.out::println); ? -
// add concurrency (using a Scheduler) Observable.create(subscriber -> { try
{ subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .subscribe(System.out::println); ? Learn more about parameterized concurrency and virtual time with Rx Schedulers at https://github.com/ReactiveX/RxJava/wiki/Scheduler -
// add operator Observable.create(subscriber -> { try { subscriber.onNext(getData()); subscriber.onCompleted();
} catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .map(data -> data + " --> at " + System.currentTimeMillis()) .subscribe(System.out::println); ? -
// add error handling Observable.create(subscriber -> { try { subscriber.onNext(getData());
subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .map(data -> data + " --> at " + System.currentTimeMillis()) .onErrorResumeNext(e -> Observable.just("Fallback Data")) .subscribe(System.out::println); -
// infinite Observable.create(subscriber -> { int i=0; while(!subscriber.isUnsubscribed()) { subscriber.onNext(i++);
} }).subscribe(System.out::println); Note: No backpressure support here. See Observable.from(Iterable) or Observable.range() for actual implementations -
Hot Cold emits whether you’re ready or not examples mouse
and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data }) -
Hot Cold emits whether you’re ready or not examples mouse
and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data }) flow control flow control & backpressure -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
Observable<R> b = Observable<T>.flatMap({ T t -‐>
Observable<R> r = ... transform t ... return r; }) flatMap -
Observable<R> b = Observable<T>.flatMap({ T t -‐>
Observable<R> r = ... transform t ... return r; }) flatMap -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
Observable.zip(a, b, (a, b) -‐> {
... operate on values from both a & b ... return Arrays.asList(a, b); }) zip { ( , ) } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
class VideoService { def VideoList getPersonalizedListOfMovies(userId);
def VideoBookmark getBookmark(userId, videoId); def VideoRating getRating(userId, videoId); def VideoMetadata getMetadata(videoId); } class VideoService { def Observable<VideoList> getPersonalizedListOfMovies(userId); def Observable<VideoBookmark> getBookmark(userId, videoId); def Observable<VideoRating> getRating(userId, videoId); def Observable<VideoMetadata> getMetadata(videoId); } ... create an observable api: instead of a blocking api ... -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } 1 2 3 4 5 6 -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } 1 2 3 4 5 6 -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request
User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } -
Decouples Consumption from Production // first request User object return
getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) -
// first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { //
then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) Decouples Consumption from Production -
// first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { //
then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) Decouples Consumption from Production -
Decouples Consumption from Production Event Loop API Request API Request
API Request API Request API Request Dependency REST API new Command().observe() new Command(). observe() new Command(). observe() new Command(). observe() new Command(). observe() Collapser -
// first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { //
then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) ~5 network calls (#3 and #4 may result in more due to windowing) 1 2 3 4 5 -
Clear API Communicates Potential Cost class VideoService {
def Observable<VideoList> getPersonalizedListOfMovies(userId); def Observable<VideoBookmark> getBookmark(userId, videoId); def Observable<VideoRating> getRating(userId, videoId); def Observable<VideoMetadata> getMetadata(videoId); } -
class VideoService { def Observable<VideoList> getPersonalizedListOfMovies(userId);
def Observable<VideoBookmark> getBookmark(userId, videoId); def Observable<VideoRating> getRating(userId, videoId); def Observable<VideoMetadata> getMetadata(videoId); } Implementation Can Differ BIO Network Call Local Cache Collapsed Network Call -
class VideoService { def Observable<VideoList> getPersonalizedListOfMovies(userId);
def Observable<VideoBookmark> getBookmark(userId, videoId); def Observable<VideoRating> getRating(userId, videoId); def Observable<VideoMetadata> getMetadata(videoId); } Implementation Can Differ and Change Local Cache Collapsed Network Call Collapsed Network Call BIO NIO Network Call -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).onErrorResumeNext(throwable -> { return
Observable.just("fallback value"); }).subscribe(System.out::println); -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).onErrorReturn(throwable -> { return
"fallback value"; }).subscribe(System.out::println); -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return
attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace()); -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return
attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace()); -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return
attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace()); -
Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return
attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace()); -
Concurrency an Observable is sequential (no concurrent emissions) scheduling and
combining Observables enables concurrency while retaining sequential emission -
// merging async Observables allows each // to execute concurrently
Observable.merge(getDataAsync(1), getDataAsync(2)) merge -
// concurrently fetch data for 5 items Observable.range(0, 5).flatMap(i ->
{ return getDataAsync(i); }) -
Observable.range(0, 5000).window(500).flatMap(work -> { return work.observeOn(Schedulers.computation()) .map(item -> { //
simulate computational work try { Thread.sleep(1); } catch (Exception e) {} return item + " processed " + Thread.currentThread(); }); }) -
Observable.range(0, 5000).buffer(500).flatMap(is -> { return Observable.from(is).subscribeOn(Schedulers.computation()) .map(item -> { //
simulate computational work try { Thread.sleep(1); } catch (Exception e) {} return item + " processed " + Thread.currentThread(); }); }) -
Hot Cold emits whether you’re ready or not examples mouse
and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data }) flow control flow control & backpressure -
Observable.range(1, 1000000).buffer(10, TimeUnit.MILLISECONDS) .toBlocking().forEach(list -> System.out.println("batch: " + list.size())); batch:
71141 batch: 49488 batch: 141147 batch: 141432 batch: 195920 batch: 240462 batch: 160410 -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] -
/* The following will emit a buffered list as it
is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] https://gist.github.com/benjchristensen/e4524a308456f3c21c0b -
Observable.range(1, 1000000).window(50, TimeUnit.MILLISECONDS) .flatMap(window -> window.count()) .toBlocking().forEach(count -> System.out.println("num items:
" + count)); num items: 477769 num items: 155463 num items: 366768 -
Observable.range(1, 1000000).window(500000) .flatMap(window -> window.count()) .toBlocking().forEach(count -> System.out.println("num items: "
+ count)); num items: 500000 num items: 500000 -
Push (reactive) when consumer keeps up with producer. Switch to
Pull (interactive) when consumer is slow. Bound all* queues. -
*vertically, not horizontally Push (reactive) when consumer keeps up with
producer. Switch to Pull (interactive) when consumer is slow. Bound all* queues. -
Observable.from(iterable) Observable.from(0, 100000) Cold Streams emits when requested (generally at
controlled rate) examples database query web service request reading file -
Observable.from(iterable) Observable.from(0, 100000) Cold Streams emits when requested (generally at
controlled rate) examples database query web service request reading file Pull -
Reactive Pull hot receives signal *including Observables that don’t implement
reactive pull support -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) Hot Infinite Stream -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return
playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) -
final ActorSystem system = ActorSystem.create("InteropTest"); final FlowMaterializer mat = FlowMaterializer.create(system);
// RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // convert to Akka Streams Source and transform using Akka Streams ‘map’ and ‘take’ operators Source<String> stringSource = Source.from(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Akka to Rx Observable return RxReactiveStreams.toObservable(stringSource.runWith(Sink.<String> fanoutPublisher(1, 1), mat)); }); strings.toBlocking().forEach(System.out::println); system.shutdown(); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'com.typesafe.akka:akka-stream-experimental_2.11:0.10-M1' -
final ActorSystem system = ActorSystem.create("InteropTest"); final FlowMaterializer mat = FlowMaterializer.create(system);
// RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // convert to Akka Streams Source and transform using Akka Streams ‘map’ and ‘take’ operators Source<String> stringSource = Source.from(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Akka to Rx Observable return RxReactiveStreams.toObservable(stringSource.runWith(Sink.<String> fanoutPublisher(1, 1), mat)); }); strings.toBlocking().forEach(System.out::println); system.shutdown(); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'com.typesafe.akka:akka-stream-experimental_2.11:0.10-M1' -
// RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i
-> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // Convert to Reactor Stream and transform using Reactor Stream ‘map’ and ‘take’ operators Stream<String> linesStream = Streams.create(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Reactor Stream to Rx Observable return RxReactiveStreams.toObservable(linesStream); }); strings.toBlocking().forEach(System.out::println); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'org.projectreactor:reactor-core:2.0.0.M1' -
// RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i
-> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // Convert to Reactor Stream and transform using Reactor Stream ‘map’ and ‘take’ operators Stream<String> linesStream = Streams.create(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Reactor Stream to Rx Observable return RxReactiveStreams.toObservable(linesStream); }); strings.toBlocking().forEach(System.out::println); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'org.projectreactor:reactor-core:2.0.0.M1' -
compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'io.ratpack:ratpack-rx:0.9.10' try { RatpackServer server
= EmbeddedApp.fromHandler(ctx -> { Observable<String> o1 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "A " + i; }); Observable<String> o2 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "B " + i; }); Observable<String> o = Observable.merge(o1, o2); ctx.render( ServerSentEvents.serverSentEvents(RxReactiveStreams.toPublisher(o), e -> e.event("counter").data("event " + e.getItem())) ); }).getServer(); server.start(); System.out.println("Port: " + server.getBindPort()); } catch (Exception e) { e.printStackTrace(); } -
compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'io.ratpack:ratpack-rx:0.9.10' try { RatpackServer server
= EmbeddedApp.fromHandler(ctx -> { Observable<String> o1 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "A " + i; }); Observable<String> o2 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "B " + i; }); Observable<String> o = Observable.merge(o1, o2); ctx.render( ServerSentEvents.serverSentEvents(RxReactiveStreams.toPublisher(o), e -> e.event("counter").data("event " + e.getItem())) ); }).getServer(); server.start(); System.out.println("Port: " + server.getBindPort()); } catch (Exception e) { e.printStackTrace(); } -
Concurrency and async are non-trivial. Rx doesn’t trivialize it. Rx
is powerful and rewards those who go through the learning curve. -
Single Multiple Sync T getData() Iterable<T> getData() Stream<T> getData() Async
Future<T> getData() Observable<T> getData() -
Reactive Programming in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html Optimizing
the Netflix API http://techblog.netflix.com/2013/01/optimizing-netflix-api.html Reactive Extensions (Rx) http://www.reactivex.io Reactive Streams https://github.com/reactive-streams/reactive-streams Ben Christensen @benjchristensen RxJava https://github.com/ReactiveX/RxJava @RxJava RxJS http://reactive-extensions.github.io/RxJS/ @ReactiveX jobs.netflix.com