Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Clojure/conj 2015: From REST To CQRS With Cloju...

Clojure/conj 2015: From REST To CQRS With Clojure, Kafka, & Datomic

Abstract: http://clojure-conj.org/speakers#bcalderwood
Video: https://www.youtube.com/watch?v=qDNPQo9UmJA

Have you ever hit a wall with REST? Does modeling your problem domain into CRUD-able entities feel like fitting a square peg into a round hole? Have you ever tried implementing a PATCH request (without going crazy), or debated PUT vs. POST for resource updates?

An alternative to CRUD-flavored REST is CQRS (Command and Query Responsibility Segregation). In this talk, I'll discuss the advantages and trade-offs of CQRS (and its cousin, Event Sourcing), and demonstrate them using an implementation of CQRS based on Clojure microservices, Kafka, and Datomic. These advantages include:

Operational simplicity, flexibility, and extensibility
Applying Clojure's sound model of state transitions to the resources exposed by your web application
Simplified client implementation
BONUS: We'll even be able to keep the best parts of REST!

I'll also discuss how this technique dovetails nicely with other good ideas in the Clojure community like the design of Datomic and the ideas behind Om.Next.

Bobby Calderwood

November 18, 2015
Tweet

More Decks by Bobby Calderwood

Other Decks in Programming

Transcript

  1. Who am I? • Member of the Tech Fellows team

    at • Formerly worked on the Product team at Cognitect, building and supporting Datomic with Rich, Stu, and team • Have worked at various startups and large enterprises in both business and technical roles [email protected] @bobbycalderwood https://github.com/bobby
  2. How to proceed? • Define Problem • Essential REST •

    CRUD-flavored REST • CQRS and Event Sourcing • Design • Demo • Assess benefits and trade-offs
  3. How not to proceed • This talk is NOT a

    repudiation of REST! • https://www.pandastrike.com/posts/20151015- rest-vs-relay • http://blog.steveklabnik.com/posts/2011-07-03- nobody-understands-rest-or-http • This talk does NOT describe the only possible implementation of CQRS and Event Sourcing via HTTP
  4. Goal • Provide valuable informational and transactional services via Mobile

    and Web software • To lots of customers • With excellent user experience • Securely and in compliance with regulations • Can easily experiment, enhance, maintain, and operate
  5. Organizational Scale • Large organization • Many product dev teams

    across several vertical LOBs • Horizontal dev and ops teams for shared services • Currently, deploying a customer-facing application involves development of client (mobile or single- page JS app) and supporting HTTP API services
  6. HTTP API Proliferation • Some externally-facing, to power customer UIs

    • Some partner-facing • Some internally-facing, consuming other APIs • DevOps only goes so far to taming this complexity!
  7. Teams Reinvent Wheels • API development: • Distributed systems problems:

    HTTP comms, failure modes, timeouts, retries, etc. • Auth(n|z), security • Audit, regulatory, compliance, metrics, monitoring • Database, cache • DevOps automation to build, test, deploy, operate
  8. Teams Reinvent Wheels • UI development (JS, iOS, Android): •

    How to talk to various endpoints/APIs • Failure modes, transactional semantics, granularity of action • Optimistic vs. Pessimistic updates
  9. Teams Reinvent Wheels • Library and code-reuse only go so

    far!!! • Internal frameworks and libraries • Shared practices and sample code repos • Communities of Practice/Excellence/Whathaveyou • All add layers of bloat, code generation, misdirection or obfuscation => complexity • None address the fundamental problem of HTTP API proliferation
  10. Ideal State • Simplify UI development for vertical teams by

    reducing number and complexity of HTTP APIs • Simplify HTTP API development for vertical teams by not doing so much of it, and handling Business Logic differently • Horizontal teams handle common functionality via shared service and infrastructure re-use (i.e. not via code or library re-use), Vertical teams experiment on value adds • Record clear picture of customer intent and interactions, for audit/compliance, replay, etc.
  11. The Good Parts • Clear communication semantics for big distributed

    systems in low-trust, low-coordination context • Proxying/caching of various types of requests over unknown network path • Clear (enough) error codes for when requests fail (most of the time) • Loose coupling and robustness principle • Sometimes even Conditional requests via Expiration, ETags, etc. (but not often)
  12. The Good Parts • Self-describing/documenting for ease of consumption, discovery,

    navigation • HTML representation lets humans (possible client implementors) learn/discover resources quickly • Sometimes even HATEOS (but not often)
  13. The Good Parts • Data-oriented • Requests and responses are

    data, in various representation formats, languages, and encodings • Sometimes even Content Negotiation (but not often)
  14. The Bad Parts • “To the person with a hammer…”

    proliferation • Tooling and “pop-culture” around REST leads to bad implementations
  15. The Good Parts • My language has tools for that!

    • Java — Jersey/Spring(Boot)/Dropwizard • Ruby — Rails/Sinatra • JavaScript — Node.js/Express/Loopback • Python — Django/Tastypie/Webmachine • Clojure — Ring/Pedestal/Liberator/Yada • Scala — Play/Lift/Scalatra
  16. The Good Parts • Widely supported and Understood • Lots

    of docs, examples, etc. • Easy to hire for • On-ramp, guardrails for less experienced developers • Tools like Swagger, etc. for specifying, documenting, validating
  17. All the cool kids are doing it! • It’s Agile™

    • Easy on ramp tooling makes teams feel productive vs. straw-man alternatives like SOAP et. al. • Easy to get something (poorly specified) into “production” • Microservices! Containers! DevOps! Cloud! • If it’s good enough for NetflixTwitterFacebook…
  18. All the cool kids are doing it! • REST is

    Over! http://blog.steveklabnik.com/posts/ 2012-02-23-rest-is-over
  19. Back into the Tarpit We use tools designed to eliminate

    the accidental complexity of mutability-centric object orientation in the small… • Functional languages with referential transparency • Immutable data structures • Immutable databases (Datomic)
  20. Back into the Tarpit …in order to implement a thin

    veneer over mutable objects in the large!!!
  21. The S in REST • But whereas at least Object

    Oriented languages have clear get/set semantics for attribute updates, CRUD-flavored REST has: • POST • PUT • PATCH (?!) • Tricky semantics, promote in-place mutability
  22. The Kingdom of Nouns for Distributed Systems • Jamming every

    operation into a CRUD on some resource is often unnatural, creating impedance mismatch • Complects communication semantics with those of application domain • I want to (choose one): A. “pay my credit-card bill” B. “POST a new bill payment sub-resource of the credit- card resource”
  23. The Kingdom of Nouns for Distributed Systems • Burdens clients

    with resource orchestration • Keep UI in sync via various read endpoints, index vs. show granularity, when to follow hypermedia • Create a composite action comprised of behaviors across various write endpoints • What about actions that need to transactionally modify several different resources at one time?
  24. The Kingdom of Nouns for Distributed Systems • Proliferates URL

    paths and POST/PUT/PATCH/ DELETE semantics just like Mutable Objects proliferate data access methods! • Could possibly dispatch domain behavior on • HTTP method • Header value • Path param • Query param • Some characteristic of Body
  25. Operationally rigid • The following criticisms apply to many out-of-the-

    box implementations and deployment patterns for CRUD-flavored REST… • Entire API, with all N endpoints often deploy, version, change, scale, etc. together • Scaling individual resources (from a single codebase) requires re-work • Scaling reads separate from writes is hard
  26. Obscures the story • Typical problems with mutable-state systems •

    What is the client/customer’s actual intent? • What happened to bring us to the current state? • How do we audit that sequence of events for security, compliance, or correctness? • Splunk on request logs anyone?
  27. What is CQRS? • Command (and) Query Responsibility Segregation •

    Separate interface and model for write (command) and read (query) paths in an application • In REST/HTTP terms, perhaps: • /commands for writes • /query (pull) and /updates (push, e.g. via WS or SSE) for read
  28. CQRS Baggage • CQRS in the small, i.e. within a

    single codebase • For each Entity/Object, have a read class and a write class • Complexity! Proliferation! • CQRS in the large, i.e. architecture of a distributed system • At a whole service/enterprise level, have one read endpoint and one write endpoint • Simplicity! Reduction!
  29. CQRS Baggage • CQRS in the small, i.e. within a

    single codebase • For each Entity/Object, have a read class and a write class • Complexity! Proliferation! • CQRS in the large, i.e. architecture of a distributed system • At a whole service/enterprise level, have one read endpoint and one write endpoint • Simplicity! Reduction!
  30. CQRS Baggage • Object-Oriented CQRS (esp. Java and C#) got

    all tangled up with • Domain-Driven Design • Aspect Oriented Programming • Naked Objects • OO “architecture” (again, in the small)
  31. CQRS Baggage • We functional programmers know better! • The

    Language of the System • Immutability • Values flow through system being transformed • Isolate and delay side-effects until the edges
  32. CQRS via HTTP /commands(/:id) /query POST GET (single or collection)

    GET /updates WS or SSE Clients Web Services + Business Logic Database Data Storage
  33. Why might I use CQRS? • Simplicity and de-complecting •

    Operational simplicity and agility • Client implementation simplicity and agility (Om.Next!)
  34. What is Event Sourcing? • Model and calculate a system’s

    state by storing all events that update that state • Often associated with CQRS: store all the events that come through command path
  35. CQRS+ES via HTTP /commands(/:id) /query POST GET (single or collection)

    GET /updates WS or SSE Clients Web Services + Business Logic Indexes Event Log Data Storage
  36. Datomic was CQRS+ES before it was cool • Datomic follows

    a CQRS design pattern with Event Sourcing, whence much of the goodness • Reads are cacheable and distributed among peers reading from storage • Writes are consistent, atomic, sequential, and only performed by transactor • State of database at time t is (logically) the aggregation of all prior events
  37. Why might I use Event Sourcing? • Your application models

    state transitions over time • You need to integrate with external systems, or extend existing functionality • You need a reified, immutable record of customer/client intent in domain terms!
  38. Why might I use Event Sourcing? /commands(/:id) /query POST GET

    (single or collection) GET /updates WS or SSE Clients Web Services + Business Logic Indexes Event Log Data Storage External Processes Processes! Processes! Processes!
  39. Why might I prefer CRUD- flavored REST over CQRS+ES? •

    External requirement or existing SLA for “REST” API • Legacy codebase with existing clients • Can’t afford additional operational cost • Messaging/event-tracking system in addition to a database • Microservice cardinality (i.e. deploy, manage, scale three endpoints instead of one) • Your problem domain truly is resource-oriented and CRUD-y
  40. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  41. Command • A two-tuple of action and data 1 [:create-customer

    {:first_name "John" 2 :last_name "Doe" 3 :address {:street_number "1234" 4 :street_name "Main St." 5 :city "Anytown" 6 :state "NH" 7 :zip "03755"}}]
  42. Query • A Datomic datalog query 1 [:find (pull ?e

    [*]) 2 :where [?e :customer/id]]
  43. Update • A Websocket message about updated data 1 [:tx/accepted

    {:key #uuid "5644d334-c537-40de-901e-5cdb9ede4c51" 2 :value {:offset "5" 3 :system "local-kafka" 4 :stream "pending-txes" 5 :partition 0 6 :basis-t 1039} 7 :topic "accepted-txes" 8 :partition 0 9 :offset 3}]
  44. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  45. 1 (ns cqrs.client 2 (:require-macros [cljs.core.async.macros :refer [go]]) 3 (:require

    [cljs-http.client :as http] 4 [cljs.core.async :as a] 5 [cognitect.transit :as t] 6 [taoensso.sente :as sente :refer (cb-success?)] 7 [taoensso.sente.packers.transit :as sente-transit])) 8 9 (defn- request-opts 10 [opts] 11 (assoc opts 12 :channel (a/promise-chan) 13 :headers {"accept" "application/transit+json"})) 14 15 (defn command! 16 ([action data] 17 (transact action data false)) 18 ([action data sync?] 19 (http/post "/api/v1/commands" 20 (cond-> {:transit-params [action data]} 21 true request-opts 22 sync? (assoc :query-params {:sync true}))))) 23 24 (defn- db-arg? 25 [o] 26 (and (map? o) 27 (or (contains? o :basis-t) 28 (contains? o :as-of) 29 (contains? o :history)))) 30 31 (defn q 32 [query & args] 33 (let [[db args] (if (db-arg? (first args)) 34 [(first args) (rest args)] 35 [nil args]) 36 {:keys [history as-of basis-t]} db 37 as-of (or as-of basis-t)] 38 (http/get "/api/v1/query" 39 (cond-> {:query-params {:query (pr-str query)}} 40 true request-opts 41 (seq args) (assoc :args (pr-str args)) 42 as-of (assoc-in [:query-params :as-of] as-of) 43 history (assoc-in [:query-params :history] history))))) 44 45 (defn updates 46 ([] 47 (updates 48 (a/chan 1 (comp (map :event) 49 (filter #(= :chs/recv (first %))))))) 50 ([ch] 51 (-> "/api/v1/updates" 52 (sente/make-channel-socket! 53 {:packer (sente-transit/->TransitPacker :json {} {}) 54 :type :auto}) 55 :ch-recv 56 (a/pipe ch))))
  46. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  47. 43 (defresource commands [datomic producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub] 44 :available-media-types

    ["text/html" "application/json" "application/transit+json" "application/edn"] 45 :known-content-type? #(check-content-type % ["application/json" 46 "application/transit+json" 47 "application/edn" 48 "application/x-www-form-urlencoded"]) 49 :allowed-methods #{:post :get} 50 :processable? parse-and-validate-cmd 51 52 ... 53 54 :post! 55 (fn [ctx] 56 (let [id (d/squuid) 57 sync (get-in ctx [:request :params :sync]) 58 result (if sync 59 (sync-post! producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub (::cmd ctx) id) 60 (async-post! producer pending-cmd-topic (::cmd ctx) id))] 61 {::id id 62 ::sync sync 63 ::result (dissoc result :success?) 64 ::success? (:success? result) 65 :location (str "/api/v1/commands/" id)})) 66 67 ... 68 69 )
  48. 1 (defn await-cmd-submission 2 [id ch] 3 (a/alt!! 4 ch

    ([v] (assoc v :id id)) 5 6 (a/timeout 2000) ([v] (throw (ex-info "Timeout writing to Kafka" {:id id}))))) 7 8 (defn async-post! 9 [producer pending-cmd-topic cmd id] 10 (await-cmd-submission id 11 (kafka/send! producer {:topic pending-cmd-topic 12 :key id 13 :value cmd}))) 14 15 (defn sync-post! 16 [producer pending-cmd-topic accepted-cmd-pub failed-cmd-pub cmd id] 17 (let [ach (a/promise-chan) 18 _ (a/sub accepted-cmd-pub id ach) 19 fch (a/promise-chan) 20 _ (a/sub failed-cmd-pub id fch) 21 _ (log/info ::sync-post! :pre-cmd) 22 cmdr (await-cmd-submission id 23 (kafka/send! producer {:topic pending-cmd-topic 24 :key id 25 :value cmd})) 26 _ (log/info ::sync-post! :post-cmd 27 :cmd cmd 28 :cmdr cmdr 29 :accepted-cmd-pub accepted-cmd-pub 30 :failed-cmd-pub failed-cmd-pub) 31 base {:cmd cmd 32 :pending cmdr} 33 ret (a/alt!! 34 ach ([v] (assoc base :success? true :accepted v)) 35 fch ([v] (assoc base :success? false :failed v)) 36 (a/timeout 5000) ([v] (throw (ex-info "Timeout reading from Kafka" {:cmd cmd :cmdr cmdr}))))] 37 (a/close! ach) 38 (a/unsub accepted-cmd-pub id ach) 39 (a/close! fch) 40 (a/unsub failed-cmd-pub id fch) 41 ret))
  49. 1 (defresource query [datomic] 2 :available-media-types ["text/html" "application/json" "application/transit+json" "application/edn"]

    3 :allowed-methods #{:get} 4 :processable? (fn [ctx] 5 (let [{:keys [query args as-of history]} (get-in ctx [:request :params])] 6 [(boolean query) {::query (edn/read-string query) 7 ::args (edn/read-string args) 8 ::as-of (safe-parse-int as-of) 9 ::history (= history "true")}])) 10 :handle-unprocessable-entity (fn [ctx] 11 (let [{:keys [::query ::args ::as-of ::history]} ctx 12 media-type (get-in ctx [:representation :media-type])] 13 (if (= media-type "text/html") 14 (h/layout 15 [:div.query 16 [:h2 "Invalid Query"] 17 (query-form-hiccup ctx)]) 18 {:query (pr-str query) 19 :args (pr-str args) 20 :as-of as-of 21 :history history 22 :error "invalid query"}))) 23 :handle-not-found (fn [ctx] []) 24 :exists? (fn [ctx] 25 (when-let [db (-> datomic :conn (get-database ctx))] 26 {::db db})) 27 28 :handle-ok (fn [ctx] 29 (let [{:keys [::query ::args ::as-of ::history]} ctx 30 media-type (get-in ctx [:representation :media-type]) 31 db (::db ctx) 32 result (apply d/q 33 query 34 db 35 args)] 36 (if (= media-type "text/html") 37 (h/layout 38 [:div.query 39 [:h2 "Query"] 40 (query-form-hiccup ctx)] 41 [:div.results 42 [:h2 "Results"] 43 (if (seq result) 44 [:table.results.pure-table.pure-table-bordered 45 (for [i result] 46 [:tr.result 47 (for [j i] 48 [:td.value (pr-str j)])])] 49 [:p "No results."])]) 50 {:query (pr-str query) 51 :args (pr-str args) 52 :basis-t (d/basis-t db) 53 :history history 54 :result result}))))
  50. It’s still REST! • Self documentation/description of 3 endpoints •

    HATEOS • GETS cacheable, proxy-able, etc. • Content negotiation of various representation formats, languages, encodings • Conditional requests, ETags, etc. • Response codes describe communication semantics and command status, not domain semantics
  51. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  52. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  53. 1 (ns cqrs.processing 2 (:require [datomic.api :as d] 3 [cqrs.samza

    :as samza]) 4 (:import [org.apache.samza.task MessageCollector TaskCoordinator TaskContext] 5 [org.apache.samza.config Config] 6 [org.apache.samza.system IncomingMessageEnvelope OutgoingMessageEnvelope SystemStream])) 7 8 (gen-class 9 :name cqrs.processing.TxTask 10 :prefix tx- 11 :state state 12 :init ctor 13 :implements [org.apache.samza.task.InitableTask 14 org.apache.samza.task.StreamTask]) 15 ... ;; helpers to init database connection ... ;; (defmulti action->tx) to handle various actions 61 (defn tx-process 62 [this 63 ^IncomingMessageEnvelope envelope 64 ^MessageCollector collector 65 ^TaskCoordinator coordinator] 66 (let [conn @(.state this) 67 action (.getMessage envelope) 68 id (.getKey envelope) 69 offset (.getOffset envelope) 70 ssp (.getSystemStreamPartition envelope) 71 ss (.getSystemStream ssp) 72 system (.getSystem ss) 73 stream (.getStream ss) 74 partition (-> ssp .getPartition .getPartitionId) 75 tx (-> action 76 action->tx 77 (tx-metadata id system stream partition offset)) 78 79 [success? {:keys [db-after]}] 80 (try [true @(d/transact conn tx)] 81 (catch Exception e 82 [false @(d/transact conn (-> tx 83 last 84 (assoc :tx/success? false) 85 vector))]))] 86 (samza/send-message collector 87 {:system system :stream (if success? "accepted-txes" "failed-txes")} 88 id 89 {:offset offset 90 :system system 91 :stream stream 92 :partition partition 93 :basis-t (d/basis-t db-after)})))
  54. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  55. Allocate Responsibility to Maximize Agility • Centralize and standardize the

    common, tricky, non-domain concerns • HTTP comms, distributed systems problems • Auth(n|z), security • Audit, compliance, regulatory • Decentralize and experiment with domain logic and UI
  56. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  57. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  58. /commands(/:id) /query POST GET (single or collection) GET pending-cmds /updates

    WS or SSE Canonical Datastore Cache, Index or Materialized View Transaction Processing accepted-cmds 1 2a sync Index/cache/ view update API/Data lake/ Process arbitrary topics Arbitrary command action Arbitrary command action Arbitrary action Clients Web Services Load balancing Auth(n|z) Input validation REST minus CRUD failed-cmds 2b Business Logic Command Log Datastores and Materialized Views sync cmd-status Remembering customer intent 1. Write command down, whole command on success, just metadata of command on failure/error. 2a. On successfully applied cmd, write to accepted-cmds topic 2b. On failed application of cmd, write to failed-cmds topic Query for arbitrarily nested data structures Subscribe to server-side updates to stay current Create and view commands/ transactions
  59. Allocate Responsibility to Maximize Agility • Solve the hard, non-domain

    (cost-center) concerns once • Experiment on the domain (money-making) concerns N times, with low drag • No more web API proliferation!
  60. No OLTP OLAP Split! • Analytics/Big Data/Fast Data are first-class

    • Perform analytics on live data • Improve situational awareness for leadership (Dashboards!)
  61. Simplifies Business Logic • De-couples each unit of business logic

    from • the web tier and communication concerns • every other unit of business logic • Simplify development: a Samza job is a single method interface • Simplify operations: deploy a Samza job, possibly in parallel with existing business logic!
  62. Simplifies Client Development • Abstract client communication with web tier

    into shared libraries for JS, iOS, Android • On read, UI components are demand-driven, a la Om.Next • On write, clients express intent in domain terms via commands
  63. Reifies customer intent • Command events are front and center

    in development process • Events are stored and replay-able • You can always re-construct the story of how you got here • You could even re-play using new business logic while existing system is running! • Very useful due to regulations, audit, compliance
  64. Next? • POC • Om.Next UI • Test at scale

    (load, latency) • auth(n|z) • Improve discoverability of API via hypermedia and displaying mapping of action => schema • Weigh trade-offs • Explore idea of company/service as a reactor to stream of events, vs. as database
  65. References and Inspiration • Talks with Tim Ewald and Rich

    Hickey • Datomic — http://datomic.com • Martin Kleppmann — http://www.confluent.io/blog/turning-the-database- inside-out-with-apache-samza/ • Om.Next — https://www.youtube.com/watch?v=ByNs9TG30E8 • Falcor — https://netflix.github.io/falcor/ • Relay — https://facebook.github.io/relay/ • Various CQRS musings on the web — http://martinfowler.com/bliki/ CQRS.html, https://msdn.microsoft.com/en-us/library/dn568103.aspx, https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf