Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Commander: Better Distributed Applications thro...

Commander: Better Distributed Applications through CQRS, Event Sourcing, and Immutable Logs

Talk video: https://www.youtube.com/watch?v=B1-gS0oEtYc
Reference implementation on GitHub: https://github.com/capitalone/cqrs-manager-for-distributed-reactive-services

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Perhaps instead of modeling our services like little databases, we should instead model them like reactors over immutable event streams.

REST APIs are great, but their typical implementation tightly couples various concerns that would be better separated:

* Reads (perception) with writes (action)
* Current state with historical narrative
* Business logic with HTTP request processing, and with operational concerns like metrics and monitoring

Commander is:

1. A pattern for writing distributed applications that de-couples these concerns using ideas from CQRS and Event Sourcing, thereby alleviating common frustrations with CRUD-flavored REST
2. An implementation of the core component in this pattern using Clojure in the HTTP layer, using Java with the new Kafka Streams library in the event stream processing layer, and using Datomic for persistence

The Commander pattern imposes a clear separation of action from perception, and uses immutable values conveyed by Kafka and the Kafka Streams library to separate business logic from HTTP request processing, all while preserving the historical narrative of the entire event stream.

In this talk, I'll discuss the benefits and tradeoffs of applying this pattern in an enterprise context, and both demonstrate and open-source my implementation.

Bobby Calderwood

September 17, 2016
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. Hi, I’m Bobby I’m on the Technology Fellow’s team at

    I dislike accidental complexity [email protected] @bobbycalderwood https://github.com/bobby
  2. Problem Space • Provide valuable informational and transactional services via

    Mobile and Web software • To lots of customers, with excellent user experience • Securely and in compliance with regulations • With ability to easily enhance, experiment, monitor, maintain, and operate • By many participants within a large organization
  3. Big Ideas • Immutability is central to information systems •

    Data language of system >> Programming language of components • Action and perception are not the same, and immutability facilitates their separation • Businesses services are not databases, they’re event stream reactors • Cross-cutting concerns must be satisfied in the presence of Conway’s Law
  4. Problematic Architecture /[resource](/:id) Database Logic Analytics Web Services Business Logic

    Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  5. Data Loss by Design /[resource](/:id) Database Logic Analytics Web Services

    Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  6. What is system language? /[resource](/:id) Database Logic Analytics Web Services

    Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  7. Writes Tied to Reads /[resource](/:id) Database Logic Analytics Web Services

    Business Logic Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  8. Database Leaking /[resource](/:id) Database Logic Analytics Web Services Business Logic

    Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  9. –Melvin Conway “organizations which design systems ... are constrained to

    produce designs which are copies of the communication structures of these organizations”
  10. Cross-cutting Concerns? /[resource](/:id) Database Logic Analytics Web Services Business Logic

    Databases and Services Operations Metrics/Monitoring Security/Compliance Audit Afterthoughts Other side-effects
  11. Commander • A better architecture for APIs and services using

    REST + Immutable Event Log + Reactive Event Stream Processing • The write-handling component of that architecture, my implementation is in Clojure
  12. Commander Architecture • Several categories of microservices with structured interactions

    among them • REST + CQRS + Event Sourcing + Reactive Event Stream Processing
  13. Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  14. Commander Component /commands(/:id) /[query-apis] commands /updates Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  15. Immutable Data Log /commands(/:id) /[query-apis] commands /updates Read- optimized View

    Command Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  16. An Event {"id": "d435ed18-4ff7-4cae-a21b-3adb7b06fe58", "parent": "33bb75db-6e13-48ee-8a54-b3976d3d065b", "action": "money-transferred", "data": {"id":

    "a6b903f6-0b9c-4c5b-95fa-afd4cc3bf938", "from_account": "12345", "to_account": "54321", "amount": 10000}, "timestamp": "2016-05-20T14:33:28.904-00:00"}
  17. 1 Log => n Data Views Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs events Business Logic Event Log Datastores and Materialized Views “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit HDFS Elastic Search Domain- specific
  18. Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  19. Primary Team Provides /commands(/:id) /[query-apis] commands /updates Read- optimized View

    Command Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  20. Enterprise Provides /commands(/:id) /[query-apis] commands /updates Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  21. Why Commander Component? • Single writer to commands topic •

    Ensuring schema conformance • Indexing all Commands and Events for reads and server-push • Provides optional illusion of synchrony to clients
  22. What is Kafka? • Apache Kafka is publish-subscribe messaging rethought

    as a distributed commit log • But it’s not really about messaging, that’s just the interface • Logs > Messages for my domain • It provides distributed, immutable logs!
  23. What is Kafka Streams? • A Java library for building

    streaming applications on top of Kafka, lives in your application • Low-level API for building topologies of processors, streams, and tables • High-level DSL for common patterns like filter, map, aggregations, joins, stateful and stateless processing • Nice operational characteristics (low latency, elastic, fault-tolerant)
  24. How to use Kafka Streams within Commander • Implement Command

    Processor • Implement Event consumers and producers • Provide local state management as backend for APIs
  25. Commander Architecture /commands(/:id) /[query-apis] commands /updates Read- optimized View Command

    Processing Analytics [other topics] Arbitrary command action Arbitrary command action Microconsume rs Web Services Load balancing Auth(n|z) Input validation events Business Logic Event Log Datastores and Materialized Views sync “Ledger” of events Operations Metrics/Monitoring Security/Compliance Audit User-facing APIs Third-party Partners WS/SSE audit
  26. KStreamBuilder builder = new KStreamBuilder(); KStream<UUID, Map> commands = builder.stream(commandsTopic);

    KStream<UUID, Map> customerEvents = commands .filter((id, command) -> command.get("action") .equals("create-customer")) .map((id, command) -> { Map userEvent = new HashMap(command); userEvent.put("action", "customer-created"); userEvent.put("parent", id); Map userValue = (Map) userEvent.get("data"); userValue.put("id", UUID.randomUUID()); return new KeyValue<>(UUID.randomUUID(), userEvent); }).through(eventsTopic); KStream<UUID, Map> customers = customerEvents .map((id, event) -> { Map customer = (Map) event.get("data"); UUID customerId = (UUID) customer.get("id"); return new KeyValue<UUID, Map>(customerId, customer); }); customers.through(customersTopic); StateStoreSupplier store = Stores.create("Customers") .persistent() .build(); builder.addStateStore(store); customers.process(customerStore, "Customers"); this.kafkaStreams = new KafkaStreams(builder, kafkaStreamsConfig); this.kafkaStreams.start();
  27. public class CustomerStore implements Processor<UUID, Map> { private KeyValueStore<UUID, Map>

    store; public List<Customer> getCustomers() { List<Customer> customers = new ArrayList<>(); KeyValueIterator<UUID, Map> iterator = store.all(); while (iterator.hasNext()) { KeyValue<UUID, Map> entry = iterator.next(); customers.add(new Customer(entry.value)); } iterator.close(); return customers; } public Customer getCustomer(UUID id) { return new Customer(store.get(id)); } @Override public void init(ProcessorContext processorContext) { this.store = (KeyValueStore<UUID, Map>) context.getStateStore("Customers"); } @Override public void process(UUID uuid, Map map) { store.put(uuid, map); } @Override public void punctuate(long l) {} @Override public void close() {} }
  28. Summary • Capture customer intent and business events as immutable

    data in domain language • From these action streams, services implement their own functionality in this common lingua franca • building many independent data views • reactively • without temporal or organizational coordination
  29. Giant Shoulders • Immutability: Rich Hickey, Stu Halloway • CQRS:

    Udi Dahan, Martin Fowler, Chris Richardson • Kafka Event Stream Reactors: Neha Narkhede, Jay Kreps, Martin Kleppmann • Organization and Management: Mel Conway, Eliyahu Goldratt, Gene Kim, Michael Nygard
  30. References • https://tinyurl.com/capital-one-cmdr • http://www.datomic.com/ • http://blog.cognitect.com/?tag=NewNormal+Series • http://www.confluent.io/blog/event-sourcing-cqrs-stream- processing-apache-kafka-whats-connection/

    • https://engineering.linkedin.com/distributed-systems/log-what- every-software-engineer-should-know-about-real-time-datas- unifying • https://www.infoq.com/presentations/Value-Values