Clojure/conj 2015: From REST To CQRS With Clojure, Kafka, & Datomic

Abstract: http://clojure-conj.org/speakers#bcalderwood
Video: https://www.youtube.com/watch?v=qDNPQo9UmJA

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Have you ever tried implementing a PATCH request (without going crazy), or debated PUT vs. POST for resource updates?

An alternative to CRUD-flavored REST is CQRS (Command and Query Responsibility Segregation). In this talk, I'll discuss the advantages and trade-offs of CQRS (and its cousin, Event Sourcing), and demonstrate them using an implementation of CQRS based on Clojure microservices, Kafka, and Datomic. These advantages include:

Operational simplicity, flexibility, and extensibility
Applying Clojure's sound model of state transitions to the resources exposed by your web application
Simplified client implementation
BONUS: We'll even be able to keep the best parts of REST!

I'll also discuss how this technique dovetails nicely with other good ideas in the Clojure community like the design of Datomic and the ideas behind Om.Next.

Transcript

1. From REST to CQRS with Clojure, Kafka (and Samza), and Datomic
2. Who am I? • Member of the Tech Fellows team at • Formerly worked on the Product team at Cognitect, building and supporting Datomic with Rich, Stu, and team • Have worked at various startups and large enterprises in both business and technical roles bobby.calderwood@capitalone.com @bobbycalderwood https://github.com/bobby
3. How to proceed? • Define Problem • Essential REST • CRUD-flavored REST • CQRS and Event Sourcing • Design • Demo • Assess benefits and trade-offs
4. How not to proceed • This talk is NOT a repudiation of REST! • https://www.pandastrike.com/posts/20151015- rest-vs-relay • http://blog.steveklabnik.com/posts/2011-07-03- nobody-understands-rest-or-http • This talk does NOT describe the only possible implementation of CQRS and Event Sourcing via HTTP
5. Goal • Provide valuable informational and transactional services via Mobile and Web software • To lots of customers • With excellent user experience • Securely and in compliance with regulations • Can easily experiment, enhance, maintain, and operate
6. Problems
7. Organizational Scale • Large organization • Many product dev teams across several vertical LOBs • Horizontal dev and ops teams for shared services • Currently, deploying a customer-facing application involves development of client (mobile or single- page JS app) and supporting HTTP API services
8. HTTP API Proliferation • Some externally-facing, to power customer UIs • Some partner-facing • Some internally-facing, consuming other APIs • DevOps only goes so far to taming this complexity!
9. Teams Reinvent Wheels • API development: • Distributed systems problems: HTTP comms, failure modes, timeouts, retries, etc. • Auth(n|z), security • Audit, regulatory, compliance, metrics, monitoring • Database, cache • DevOps automation to build, test, deploy, operate
10. Teams Reinvent Wheels • UI development (JS, iOS, Android): • How to talk to various endpoints/APIs • Failure modes, transactional semantics, granularity of action • Optimistic vs. Pessimistic updates
11. Teams Reinvent Wheels • Library and code-reuse only go so far!!! • Internal frameworks and libraries • Shared practices and sample code repos • Communities of Practice/Excellence/Whathaveyou • All add layers of bloat, code generation, misdirection or obfuscation => complexity • None address the fundamental problem of HTTP API proliferation
12. Ideal State • Simplify UI development for vertical teams by reducing number and complexity of HTTP APIs • Simplify HTTP API development for vertical teams by not doing so much of it, and handling Business Logic differently • Horizontal teams handle common functionality via shared service and infrastructure re-use (i.e. not via code or library re-use), Vertical teams experiment on value adds • Record clear picture of customer intent and interactions, for audit/compliance, replay, etc.
13. Essential REST
14. The Good Parts • Clear communication semantics for big distributed systems in low-trust, low-coordination context • Proxying/caching of various types of requests over unknown network path • Clear (enough) error codes for when requests fail (most of the time) • Loose coupling and robustness principle • Sometimes even Conditional requests via Expiration, ETags, etc. (but not often)
15. The Good Parts • Self-describing/documenting for ease of consumption, discovery, navigation • HTML representation lets humans (possible client implementors) learn/discover resources quickly • Sometimes even HATEOS (but not often)
16. The Good Parts • Data-oriented • Requests and responses are data, in various representation formats, languages, and encodings • Sometimes even Content Negotiation (but not often)
17. The Bad Parts • “To the person with a hammer…” proliferation • Tooling and “pop-culture” around REST leads to bad implementations
18. CRUD-flavored REST
19. The Good Parts • My language has tools for that! • Java — Jersey/Spring(Boot)/Dropwizard • Ruby — Rails/Sinatra • JavaScript — Node.js/Express/Loopback • Python — Django/Tastypie/Webmachine • Clojure — Ring/Pedestal/Liberator/Yada • Scala — Play/Lift/Scalatra
20. The Good Parts • Widely supported and Understood • Lots of docs, examples, etc. • Easy to hire for • On-ramp, guardrails for less experienced developers • Tools like Swagger, etc. for specifying, documenting, validating
21. All the cool kids are doing it! • It’s Agile™ • Easy on ramp tooling makes teams feel productive vs. straw-man alternatives like SOAP et. al. • Easy to get something (poorly specified) into “production” • Microservices! Containers! DevOps! Cloud! • If it’s good enough for NetflixTwitterFacebook…
22. The Bad Parts
23. All the cool kids are doing it! • REST is Over! http://blog.steveklabnik.com/posts/ 2012-02-23-rest-is-over
24. Back into the Tarpit We use tools designed to eliminate the accidental complexity of mutability-centric object orientation in the small… • Functional languages with referential transparency • Immutable data structures • Immutable databases (Datomic)
25. Back into the Tarpit
26. Back into the Tarpit …in order to implement a thin veneer over mutable objects in the large!!!
27. Back into the Tarpit
28. The S in REST • But whereas at least Object Oriented languages have clear get/set semantics for attribute updates, CRUD-flavored REST has: • POST • PUT • PATCH (?!) • Tricky semantics, promote in-place mutability
29. The Kingdom of Nouns for Distributed Systems • Jamming every operation into a CRUD on some resource is often unnatural, creating impedance mismatch • Complects communication semantics with those of application domain • I want to (choose one): A. “pay my credit-card bill” B. “POST a new bill payment sub-resource of the credit- card resource”
30. The Kingdom of Nouns for Distributed Systems • Burdens clients with resource orchestration • Keep UI in sync via various read endpoints, index vs. show granularity, when to follow hypermedia • Create a composite action comprised of behaviors across various write endpoints • What about actions that need to transactionally modify several different resources at one time?
31. The Kingdom of Nouns for Distributed Systems • Proliferates URL paths and POST/PUT/PATCH/ DELETE semantics just like Mutable Objects proliferate data access methods! • Could possibly dispatch domain behavior on • HTTP method • Header value • Path param • Query param • Some characteristic of Body
32. Operationally rigid • The following criticisms apply to many out-of-the- box implementations and deployment patterns for CRUD-flavored REST… • Entire API, with all N endpoints often deploy, version, change, scale, etc. together • Scaling individual resources (from a single codebase) requires re-work • Scaling reads separate from writes is hard
33. Hard to extend or integrate /api/v1/resource HTTP API Database Data Storage Integrate here? Or here?!?!?
34. Obscures the story • Typical problems with mutable-state systems • What is the client/customer’s actual intent? • What happened to bring us to the current state? • How do we audit that sequence of events for security, compliance, or correctness? • Splunk on request logs anyone?
35. CQRS and Event Sourcing
36. What is CQRS? • Command (and) Query Responsibility Segregation • Separate interface and model for write (command) and read (query) paths in an application • In REST/HTTP terms, perhaps: • /commands for writes • /query (pull) and /updates (push, e.g. via WS or SSE) for read
37. CQRS Baggage • CQRS in the small, i.e. within a single codebase • For each Entity/Object, have a read class and a write class • Complexity! Proliferation! • CQRS in the large, i.e. architecture of a distributed system • At a whole service/enterprise level, have one read endpoint and one write endpoint • Simplicity! Reduction!
38. CQRS Baggage • CQRS in the small, i.e. within a single codebase • For each Entity/Object, have a read class and a write class • Complexity! Proliferation! • CQRS in the large, i.e. architecture of a distributed system • At a whole service/enterprise level, have one read endpoint and one write endpoint • Simplicity! Reduction!
39. CQRS Baggage • Object-Oriented CQRS (esp. Java and C#) got all tangled up with • Domain-Driven Design • Aspect Oriented Programming • Naked Objects • OO “architecture” (again, in the small)
40. CQRS Baggage • We functional programmers know better! • The Language of the System • Immutability • Values flow through system being transformed • Isolate and delay side-effects until the edges
41. CQRS via HTTP /commands(/:id) /query POST GET (single or collection) GET /updates WS or SSE Clients Web Services + Business Logic Database Data Storage
42. Why might I use CQRS? • Simplicity and de-complecting • Operational simplicity and agility • Client implementation simplicity and agility (Om.Next!)
43. What is Event Sourcing? • Model and calculate a system’s state by storing all events that update that state • Often associated with CQRS: store all the events that come through command path
44. CQRS+ES via HTTP /commands(/:id) /query POST GET (single or collection) GET /updates WS or SSE Clients Web Services + Business Logic Indexes Event Log Data Storage
45. Datomic was CQRS+ES before it was cool • Datomic follows a CQRS design pattern with Event Sourcing, whence much of the goodness • Reads are cacheable and distributed among peers reading from storage • Writes are consistent, atomic, sequential, and only performed by transactor • State of database at time t is (logically) the aggregation of all prior events
46. Why might I use Event Sourcing? • Your application models state transitions over time • You need to integrate with external systems, or extend existing functionality • You need a reified, immutable record of customer/client intent in domain terms!
47. Why might I use Event Sourcing? /commands(/:id) /query POST GET (single or collection) GET /updates WS or SSE Clients Web Services + Business Logic Indexes Event Log Data Storage External Processes Processes! Processes! Processes!
48. Why might I prefer CRUD- flavored REST over CQRS+ES? • External requirement or existing SLA for “REST” API • Legacy codebase with existing clients • Can’t afford additional operational cost • Messaging/event-tracking system in addition to a database • Microservice cardinality (i.e. deploy, manage, scale three endpoints instead of one) • Your problem domain truly is resource-oriented and CRUD-y
49. Design
50. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
51. Command • A two-tuple of action and data 1 [:create-customer {:first_name "John" 2 :last_name "Doe" 3 :address {:street_number "1234" 4 :street_name "Main St." 5 :city "Anytown" 6 :state "NH" 7 :zip "03755"}}]
52. Query • A Datomic datalog query 1 [:find (pull ?e [*]) 2 :where [?e :customer/id]]
53. Update • A Websocket message about updated data 1 [:tx/accepted {:key #uuid "5644d334-c537-40de-901e-5cdb9ede4c51" 2 :value {:offset "5" 3 :system "local-kafka" 4 :stream "pending-txes" 5 :partition 0 6 :basis-t 1039} 7 :topic "accepted-txes" 8 :partition 0 9 :offset 3}]
54. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
55. 1 (ns cqrs.client 2 (:require-macros [cljs.core.async.macros :refer [go]]) 3 (:require [cljs-http.client :as http] 4 [cljs.core.async :as a] 5 [cognitect.transit :as t] 6 [taoensso.sente :as sente :refer (cb-success?)] 7 [taoensso.sente.packers.transit :as sente-transit])) 8 9 (defn- request-opts 10 [opts] 11 (assoc opts 12 :channel (a/promise-chan) 13 :headers {"accept" "application/transit+json"})) 14 15 (defn command! 16 ([action data] 17 (transact action data false)) 18 ([action data sync?] 19 (http/post "/api/v1/commands" 20 (cond-> {:transit-params [action data]} 21 true request-opts 22 sync? (assoc :query-params {:sync true}))))) 23 24 (defn- db-arg? 25 [o] 26 (and (map? o) 27 (or (contains? o :basis-t) 28 (contains? o :as-of) 29 (contains? o :history)))) 30 31 (defn q 32 [query & args] 33 (let [[db args] (if (db-arg? (first args)) 34 [(first args) (rest args)] 35 [nil args]) 36 {:keys [history as-of basis-t]} db 37 as-of (or as-of basis-t)] 38 (http/get "/api/v1/query" 39 (cond-> {:query-params {:query (pr-str query)}} 40 true request-opts 41 (seq args) (assoc :args (pr-str args)) 42 as-of (assoc-in [:query-params :as-of] as-of) 43 history (assoc-in [:query-params :history] history))))) 44 45 (defn updates 46 ([] 47 (updates 48 (a/chan 1 (comp (map :event) 49 (filter #(= :chs/recv (first %))))))) 50 ([ch] 51 (-> "/api/v1/updates" 52 (sente/make-channel-socket! 53 {:packer (sente-transit/->TransitPacker :json {} {}) 54 :type :auto}) 55 :ch-recv 56 (a/pipe ch))))
56. Om.Next!
57. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
58. 43 (defresource commands [datomic producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub] 44 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"] 45 :known-content-type? #(check-content-type % ["application/json" 46 "application/transit+json" 47 "application/edn" 48 "application/x-www-form-urlencoded"]) 49 :allowed-methods #{:post :get} 50 :processable? parse-and-validate-cmd 51 52 ... 53 54 :post! 55 (fn [ctx] 56 (let [id (d/squuid) 57 sync (get-in ctx [:request :params :sync]) 58 result (if sync 59 (sync-post! producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub (::cmd ctx) id) 60 (async-post! producer pending-cmd-topic (::cmd ctx) id))] 61 {::id id 62 ::sync sync 63 ::result (dissoc result :success?) 64 ::success? (:success? result) 65 :location (str "/api/v1/commands/" id)})) 66 67 ... 68 69 )
59. 1 (defn await-cmd-submission 2 [id ch] 3 (a/alt!! 4 ch ([v] (assoc v :id id)) 5 6 (a/timeout 2000) ([v] (throw (ex-info "Timeout writing to Kafka" {:id id}))))) 7 8 (defn async-post! 9 [producer pending-cmd-topic cmd id] 10 (await-cmd-submission id 11 (kafka/send! producer {:topic pending-cmd-topic 12 :key id 13 :value cmd}))) 14 15 (defn sync-post! 16 [producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub cmd id] 17 (let [ach (a/promise-chan) 18 _ (a/sub accepted-cmd-pub id ach) 19 fch (a/promise-chan) 20 _ (a/sub failed-cmd-pub id fch) 21 _ (log/info ::sync-post! :pre-cmd) 22 cmdr (await-cmd-submission id 23 (kafka/send! producer {:topic pending-cmd-topic 24 :key id 25 :value cmd})) 26 _ (log/info ::sync-post! :post-cmd 27 :cmd cmd 28 :cmdr cmdr 29 :accepted-cmd-pub accepted-cmd-pub 30 :failed-cmd-pub failed-cmd-pub) 31 base {:cmd cmd 32 :pending cmdr} 33 ret (a/alt!! 34 ach ([v] (assoc base :success? true :accepted v)) 35 fch ([v] (assoc base :success? false :failed v)) 36 (a/timeout 5000) ([v] (throw (ex-info "Timeout reading from Kafka" {:cmd cmd :cmdr cmdr}))))] 37 (a/close! ach) 38 (a/unsub accepted-cmd-pub id ach) 39 (a/close! fch) 40 (a/unsub failed-cmd-pub id fch) 41 ret))
60. 1 (defresource query [datomic] 2 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"] 3 :allowed-methods #{:get} 4 :processable? (fn [ctx] 5 (let [{:keys [query args as-of history]} (get-in ctx [:request :params])] 6 [(boolean query) {::query (edn/read-string query) 7 ::args (edn/read-string args) 8 ::as-of (safe-parse-int as-of) 9 ::history (= history "true")}])) 10 :handle-unprocessable-entity (fn [ctx] 11 (let [{:keys [::query ::args ::as-of ::history]} ctx 12 media-type (get-in ctx [:representation :media-type])] 13 (if (= media-type "text/html") 14 (h/layout 15 [:div.query 16 [:h2 "Invalid Query"] 17 (query-form-hiccup ctx)]) 18 {:query (pr-str query) 19 :args (pr-str args) 20 :as-of as-of 21 :history history 22 :error "invalid query"}))) 23 :handle-not-found (fn [ctx] []) 24 :exists? (fn [ctx] 25 (when-let [db (-> datomic :conn (get-database ctx))] 26 {::db db})) 27 28 :handle-ok (fn [ctx] 29 (let [{:keys [::query ::args ::as-of ::history]} ctx 30 media-type (get-in ctx [:representation :media-type]) 31 db (::db ctx) 32 result (apply d/q 33 query 34 db 35 args)] 36 (if (= media-type "text/html") 37 (h/layout 38 [:div.query 39 [:h2 "Query"] 40 (query-form-hiccup ctx)] 41 [:div.results 42 [:h2 "Results"] 43 (if (seq result) 44 [:table.results.pure-table.pure-table-bordered 45 (for [i result] 46 [:tr.result 47 (for [j i] 48 [:td.value (pr-str j)])])] 49 [:p "No results."])]) 50 {:query (pr-str query) 51 :args (pr-str args) 52 :basis-t (d/basis-t db) 53 :history history 54 :result result}))))
61. It’s still REST! • Self documentation/description of 3 endpoints • HATEOS • GETS cacheable, proxy-able, etc. • Content negotiation of various representation formats, languages, encodings • Conditional requests, ETags, etc. • Response codes describe communication semantics and command status, not domain semantics
62. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
63. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
64. 1 (ns cqrs.processing 2 (:require [datomic.api :as d] 3 [cqrs.samza :as samza]) 4 (:import [org.apache.samza.task MessageCollector TaskCoordinator TaskContext] 5 [org.apache.samza.config Config] 6 [org.apache.samza.system IncomingMessageEnvelope OutgoingMessageEnvelope SystemStream])) 7 8 (gen-class 9 :name cqrs.processing.TxTask 10 :prefix tx- 11 :state state 12 :init ctor 13 :implements [org.apache.samza.task.InitableTask 14 org.apache.samza.task.StreamTask]) 15 ... ;; helpers to init database connection ... ;; (defmulti action->tx) to handle various actions 61 (defn tx-process 62 [this 63 ^IncomingMessageEnvelope envelope 64 ^MessageCollector collector 65 ^TaskCoordinator coordinator] 66 (let [conn @(.state this) 67 action (.getMessage envelope) 68 id (.getKey envelope) 69 offset (.getOffset envelope) 70 ssp (.getSystemStreamPartition envelope) 71 ss (.getSystemStream ssp) 72 system (.getSystem ss) 73 stream (.getStream ss) 74 partition (-> ssp .getPartition .getPartitionId) 75 tx (-> action 76 action->tx 77 (tx-metadata id system stream partition offset)) 78 79 [success? {:keys [db-after]}] 80 (try [true @(d/transact conn tx)] 81 (catch Exception e 82 [false @(d/transact conn (-> tx 83 last 84 (assoc :tx/success? false) 85 vector))]))] 86 (samza/send-message collector 87 {:system system :stream (if success? "accepted-txes" "failed-txes")} 88 id 89 {:offset offset 90 :system system 91 :stream stream 92 :partition partition 93 :basis-t (d/basis-t db-after)})))
65. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
66. Demo
67. Benefits of Design
68. Allocate Responsibility to Maximize Agility • Centralize and standardize the common, tricky, non-domain concerns • HTTP comms, distributed systems problems • Auth(n|z), security • Audit, compliance, regulatory • Decentralize and experiment with domain logic and UI
69. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
70. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
71. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
72. Allocate Responsibility to Maximize Agility • Solve the hard, non-domain (cost-center) concerns once • Experiment on the domain (money-making) concerns N times, with low drag • No more web API proliferation!
73. No OLTP OLAP Split! • Analytics/Big Data/Fast Data are first-class • Perform analytics on live data • Improve situational awareness for leadership (Dashboards!)
74. Simplifies Business Logic • De-couples each unit of business logic from • the web tier and communication concerns • every other unit of business logic • Simplify development: a Samza job is a single method interface • Simplify operations: deploy a Samza job, possibly in parallel with existing business logic!
75. Simplifies Client Development • Abstract client communication with web tier into shared libraries for JS, iOS, Android • On read, UI components are demand-driven, a la Om.Next • On write, clients express intent in domain terms via commands
76. Reifies customer intent • Command events are front and center in development process • Events are stored and replay-able • You can always re-construct the story of how you got here • You could even re-play using new business logic while existing system is running! • Very useful due to regulations, audit, compliance
77. Next? • POC • Om.Next UI • Test at scale (load, latency) • auth(n|z) • Improve discoverability of API via hypermedia and displaying mapping of action => schema • Weigh trade-offs • Explore idea of company/service as a reactor to stream of events, vs. as database
78. References and Inspiration • Talks with Tim Ewald and Rich Hickey • Datomic — http://datomic.com • Martin Kleppmann — http://www.confluent.io/blog/turning-the-database- inside-out-with-apache-samza/ • Om.Next — https://www.youtube.com/watch?v=ByNs9TG30E8 • Falcor — https://netflix.github.io/falcor/ • Relay — https://facebook.github.io/relay/ • Various CQRS musings on the web — http://martinfowler.com/bliki/ CQRS.html, https://msdn.microsoft.com/en-us/library/dn568103.aspx, https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf
79. Thank you! Questions?