Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive SoundCloud - Tackling complexity in la...

Reactive SoundCloud - Tackling complexity in large applications

In this talk I give some insights into how we rethought our app architecture from the ground up. I show where the complexity lies in our application and how we use the Reactive Extensions (Rx) to gain the upper hand back on dealing with the representation and composition of asynchronous data flows. I also introduce propeller, our storage library which fits snuggly into our reactive architecture.

Matthias Käppler

October 10, 2014
Tweet

More Decks by Matthias Käppler

Other Decks in Programming

Transcript

  1. Essential Complexity vs. Accidental Complexity title, date, 01 of 10

    No Silver Bullet — Essence and Accidents of Software Engineering, Fred Brooks, 1986
  2. Essential Complexity title, date, 01 of 10 Dealing with multiple

    asynchronous data sources in a fault tolerant manner → Propagation of change → Propagation of errors
  3. Accidental Complexity title, date, 01 of 10 Self-inflicted complexity, e.g.

    due to lack of, misuse of, or overuse of abstractions.
  4. Accidental Complexity title, date, 01 of 10 Unsuitable database abstraction

    Little support for handling async No standard event/error model Optimizing for the 1%
  5. title, date, 01 of 10 “It’s the duty of the

    architect to solve the problems inherent in essential complexity without introducing accidental complexity.” -- Neil Ford
  6. Problem Statement title, date, 01 of 10 Introduce a uniform

    model for concurrency, compositional events and error handling.
  7. Proposition title, date, 01 of 10 This can be accomplished

    using functional reactive programming.
  8. Proposition title, date, 01 of 10 Sorry, I meant functional-ish

    reactive-ish programming. “For purity we are trying to avoid using the term FRP in the context of Rx” https://twitter. com/headinthebox/status/4925832274090721 28
  9. Messaging Protocol title, date, 01 of 10 Observer<T> Observer<T> Observable<T>

    Observer<T> Observable<T> onNext(T) onError(Throwable) React Deliver / Transform Emit
  10. Example: Loading data Observable<Track> track(Urn trackUrn) { return trackFromStorage(trackUrn) .toList()

    .flatMap(syncIfEmpty(trackUrn)); } Observable<Track> fullTrack(Urn trackUrn) { return trackFromStorage(trackUrn) .zipWith(trackStorage.trackDetails(trackUrn), mergeFunction); } title, date, 01 of 10
  11. Message Passing title, date, 01 of 10 interface EventBus {

    <T> Subscription subscribe(Queue<T> queue, Observer<T> observer); <T> void publish(Queue<T> queue, T event); <T> Subject<T, T> queue(Queue<T> queue); }
  12. Message Passing title, date, 01 of 10 Receiver @Inject EventBus

    eventBus; eventBus.subscribe(EventQueue.MY_QUEUE, new Subscriber() { public void onNext(MyEvent event) { … } }); Sender @Inject EventBus eventBus; eventBus.publish(EventQueue.MY_QUEUE, new MyEvent());
  13. Traditional Storage title, date, 01 of 10 Storage Logic Presentation

    Network class Track { String title User user } class TrackStorage { void insert(Track track) Track load(long trackId) } Track Track
  14. CQRS title, date, 01 of 10 ReadStorage Logic Presentation Network

    class ApiTrack { String title User user } class TrackWriteStorage { void insert(ApiTrack track) } class TrackReadStorage { PropertySet load(long id) } WriteStorage ApiTrack PropertySet
  15. Traditional Model title, date, 01 of 10 class Track {

    private Urn urn; private String title; private User user; }
  16. Traditional Model title, date, 01 of 10 class Track {

    private Urn urn; private String title; private User user; } Track track = new Track(); track.setUrn(new Urn(“soundcloud:tracks:1”)); track.setTitle(“title”); track.setUser(new User(...));
  17. Properties title, date, 01 of 10 interface Track { Property<Urn>

    URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; }
  18. Property Sets title, date, 01 of 10 interface Track {

    Property<Urn> URN = Property.of(Urn.class); Property<String> TITLE = Property.of(String.class); Property<Urn> CREATOR = User.URN; } PropertySet track = PropertySet.from( Track.URN.bind(new Urn(“soundcloud:tracks:1”)), Track.TITLE.bind(“track title”) Track.CREATOR.bind(new Urn(“soundcloud:users:2”)) );
  19. (Type) safe & composable title, date, 01 of 10 //

    will throw in if TITLE is missing String title = track.get(Track.TITLE); // returns ‘alt’ if TITLE is missing String title = track.getOrElse(Track.TITLE, “alt”); // returns null if TITLE is missing String title = track.getOrElseNull(Track.TITLE); // combines two sets PropertySet ps3 = ps1.merge(ps2);
  20. propeller title, date, 01 of 10 Design goals → Minimal

    bias → Symmetric push and pull → Uniform error handling
  21. Query title, date, 01 of 10 PropellerDatabase propeller = new

    PropellerDatabase(sqliteDatabase); Query q = Query.from(“tracks”) .select(“title”) .whereEq(“plays”, 42) .limit(1); QueryResult result = propeller.query(q);
  22. Result Iterators title, date, 01 of 10 for (CursorReader row

    : result) { String title = row.getString(“title”); ... }
  23. Row Mapping title, date, 01 of 10 mapper = new

    ResultMapper<Track> { @Override public Track map(CursorReader row) { return new Track( row.getString(“title”) ); } } List<Track> tracks = result.toList(mapper);
  24. Result Observables title, date, 01 of 10 DatabaseScheduler scheduler =

    new DatabaseScheduler(prop, sched); Observable<Track> tracks = scheduler.scheduleQuery(query).map(mapper);
  25. What else? title, date, 01 of 10 Read Joining, filtering,

    ordering, limiting, column functions Write (bulk)insert, (bulk)upsert, update, delete, truncate Transactions Integration test helpers
  26. Summary title, date, 01 of 10 Rx reduces complexity when

    composing data sources Rx only provides the cogs, YOU build the clockwork Don’t fall victim to MacGyver, always challenge your architecture!
  27. Presentation: Fragments title, date, 01 of 10 class CommentsFragment extends

    DefaultFragment implements ReactiveListComponent<...> { @Inject CommentsOperations operations; @Inject EndlessAdapter adapter @Inject ListViewController controller; private ConnectableObservable<List<Comment>> comments; private Subscription sub = Subscriptions.empty() // life-cycle methods ... }
  28. Presentation: Fragments title, date, 01 of 10 public Observable<List<Comment>> buildObservable()

    { TrackUrn trackUrn = getArguments().getParcelable(...); this.comments = operations.comments(trackUrn) .map(toCommentViewModel) .observeOn(mainThread()) .replay(1); this.comments.subscribe(adapter); } public Subscription connectObservable() { this.sub = comments.connect(); return sub; }
  29. Presentation: Fragments title, date, 01 of 10 public void onViewCreated(View

    view, Bundle bundle) { super.onViewCreated(view, bundle); controller.connect(this, comments); } public void onDestroy() { this.sub.unsubscribe(); super.onDestroy(); }
  30. Branch by Abstraction title, date, 01 of 10 <<new component>>

    branch <<old component>> master <<old component>> origin/master
  31. master Branch by Abstraction title, date, 01 of 10 abstraction

    layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>> <<old component>>
  32. master Branch by Abstraction title, date, 01 of 10 abstraction

    layer http://martinfowler.com/bliki/BranchByAbstraction.html <<new component>>
  33. master Trunk Based Development title, date, 01 of 10 feature-B

    .isEnabled() .isEnabled() feature-A .isEnabled() feature-C
  34. Confident Code title, date, 01 of 10 Uses “null objects”

    or option types: Collections.emptyList() Optional.of()
  35. Confident Code title, date, 01 of 10 Follows a narrative

    structure that tells, not asks: 1. Gather inputs 2. Perform work 3. Deliver result 4. Handle failure