Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Introduction to Event Sourcing for Rubyists

An Introduction to Event Sourcing for Rubyists

Event sourcing is a design pattern to build applications that are domain centric and easy to extend. The pattern is based on the usage of a persistent event log which substitutes the more classical relational database model for Rails applications.

Such event log can then be used for extending your application in all sort of creative ways. For example, by synchronizing data between your microservices, trigger side effects without cluttering your models or controllers, or build data views optimized for your query needs. In this talk, I'll present the basic ideas, some of the tradeoffs and challenges you might find and how you could start experimenting with it.

Alfredo Motta

May 09, 2018
Tweet

Other Decks in Technology

Transcript

  1. MY JOURNEY SO FAR • PhD in Software Engineering (UML

    & formal methods) • Startup launching a second-hand market for daily deals (4 ppl) • Startup growing a holiday rentals website (100+ ppl) • P2P lender launching new products for car finance (200+ ppl) • Startup building a safer way to borrow (8 ppl)
  2. WHY WE READ AND WRITE FROM THE SAME TABLE? •

    Because it’s the simplest thing that makes sense? • Because it is the Rails convention? • Because we want to read our own writes? • Because we want to take advantages of database locks?
  3. A LITTLE SECRET • Business has been dealing with stale

    data forever • Overbooking of Hotel/Flights still happens today • What if some, or even all, of your data doesn't need to be consistent all the time?
  4. EVENT SOURCING PERSPECTIVE • Track changes, not state • Write

    events, read from eventually consistent projections of these events • Focus on immutability and idempotency
  5. • GIT store your current state as a change-log •

    i.e. a list of commits that represent your current state • it is ok if you and your co- worker does not have the same db at the same time GIT • Accountants never overwrite data • they add a new entry with the latest change in your account • they can always explain the reason behind your account balance Accountants
  6. You take the blue pill, the story ends. You wake

    up in your bed and believe whatever you want to believe. You take the red pill, you stay in Wonderland and I show you how deep the rabbit hole goes
  7. –Martin Fowler “Capture all changes to an application state as

    a sequence of events.” –Greg Young “A system where an Event Store is used to rebuild an object of the domain as opposed to something storing the current state.” EVENT SOURCING
  8. BUILDING BLOCKS Events happen in the past email_updated email_updated at

    2018-05-04 08:39:34 +0100 email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 with [email protected] email_updated at 2018-05-04 08:39:34 +0100 for user with ID 99 Events are immutable Events belongs to models *** Events have a payload storing what happened *** more precisely events belong to aggregates, aggregates define consistency boundaries for groups of related entities.
  9. BENEFITS • Read schemas are decoupled from event log •

    Auditability / why state is the way it is • Simple way to synchronise data between micro services
  10. SCHEDULE A MEETING a command is a request for an

    event creation a successful command generates an event entity_id entity_type event_type data 1 Meeting scheduled { organizer_id: 1, invitee_ids: [2, 3], time: 2018-05-05 09:07:18 +0100 }
  11. EVENT SCHEMA (SIMPLE) create_table 'events', id: :serial do |t| t.uuid

    'entity_id', null: false, index: true t.string 'event_type', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false end email_updated id of the user { new_email: [email protected] } . .
  12. EVENT SCHEMA (COMPLETE) create_table 'events', id: :serial do |t| t.uuid

    'entity_id', null: false, index: true t.string 'entity_type', null: false t.string 'event_type', null: false t.jsonb 'meta', null: false t.jsonb 'data', null: false t.datetime 'created_at', null: false t.int 'entity_version', null: false t.index ['entity_id', 'entity_version'], name: 'event_version_constraint', unique: true end for example, User . . protects from multiple processes sending conflicting updates for all your meta-data needs
  13. class MeetingsController < ApplicationController def create response = EventSourced::Meeting.new(SecureRandom.uuid).schedule(schedule_meeting_params) if

    response.success? # ... else # ... end end private def schedule_meeting_params params.require(:meeting_form).permit(:time, :organizer_id, :invitee_ids) end [ … ] end 1. create meeting instance 2. invoke schedule command
  14. module EventSourced class Meeting attr_reader :id, :version, :status, :attendees_count, :time

    def initialize(id) @id = id @attendees_count = 0 @version = 0 end def schedule(time:, organizer_id:, invitee_ids:) Commands::Meeting::Schedule.call( self, time: meeting_time, organizer_id: organizer_id, invitee_ids: invitee_ids ) end [ … ] end end delegate to command for detailed validation
  15. module Commands module Meeting ScheduleSchema = Dry::Validation.Schema do required(:time).value(gt?: Time.now

    + 5.minutes) required(:organizer_id).filled required(:invitee_ids).filled { each(:int?) } end class Schedule < Command def call return validate if validate.failure? event = publish_event( event_type: 'scheduled', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { time: @params[:time], organizer_id: @params[:organizer_id], invitee_ids: @params[:invitee_ids] } ) @entity.handle(event) Response.new success: true, data: { new_meeting: @entity } end end end end dry-validation gem to check the event data store the event with the appropriate data update the entity
  16. module EventSourced class Meeting [ … ] def scheduled_handler(event) params

    = event.data.symbolize_keys @time = params[:time] end def handle(event) handler = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end [ … ] end end the event handler is responsible for updating the in-memory domain object
  17. class MeetingsController < ApplicationController [ … ] def accept meeting_id

    = accept_meeting_params[:meeting_id] response = EventSourced::Meeting .get(meeting_id) .accept(user_id: accept_meeting_params[:user_id]) flash[:alert] = "Failed to accept the meeting"if response.failure? redirect_to meeting_path(meeting_id) end private [ … ] def accept_meeting_params params.require(:accept_form).permit(:meeting_id, :user_id) end end get the entity and issue the command
  18. module EventSourced class Meeting [ … ] def handle(event) handler

    = "#{event.event_type}_handler".to_sym if entity.respond_to? handler send(handler, event) @version += 1 end end class << self def get(id) entity = new(id) load_events_for(entity) entity end def load_events_for(entity) events = Event.where(entity_type: 'meeting', entity_id: entity.id) events.each { |event| entity.handle(event) } end end end end send event to handler and increment version load events from the store and applies them to the given entity
  19. WHERE ARE MY ACTIVE RECORD MODELS? • They become read-optimised

    views of your data • The single source of truth is the event log and your relational tables are a view on that log • Pro tip: you can have as many views as you want (think of reporting, search, analytics)
  20. class Meeting < ApplicationRecord has_one :organizer has_many :invitees class <<

    self def scheduled_handler(event) params = event.data.symbolize_keys create!( entity_id: event.entity_id, time: params.fetch(:time), organizer_id: params.fetch(:organizer_id), created_at: event.created_at ) end def rescheduled_handler(event) meeting = find(event.entity_id) params = event.data.symbolize_keys meeting.update_attribute(time: params.fetch(:new_time)) end [ … ] end end create a row in a relational table when a scheduled event is received update an existing entry when a rescheduled event is received
  21. IN SHORT • Event Sourcing is GIT applied to your

    data • You invoke commands to write events • Events become the single source of truth of your data • Your old active record models become your read- optimised views
  22. BENEFITS • Read schemas are decoupled from event log •

    Auditability / Why state is the way it is • Simple way to synchronise data between micro services
  23. DRAWBACKS • More complex • You will have to think

    about eventual consistency • You will have to think about idempotency • You de-rail from the Rails conventions
  24. EVENT SOURCING IN RUBY • Beyond the current state: Time

    travel to the rescue!, Armin Pašalić, wroc_love.rb 2018 • Event Sourcing: A Rails Case Study, Francis Hwang, NYC.rb 2015 • https://github.com/fhwang/event_sourced_record • https://github.com/RailsEventStore/rails_event_store • https://github.com/Sandthorn/sandthorn • https://github.com/zilverline/sequent
  25. EVENT SOURCING REFERENCES • The Many Meanings of Event-Driven Architecture,

    Martin Fowler, GOTO 2017 • Event Sourcing, Martin Fowler, YOW! Nights 2016 • Implementing Domain Driven Design, Vaugh Vernon, 2013 • Greg Young Event DDD, CQRS, Event sourcing course • http://subscriptions.viddler.com/GregYoung/ • Promo code: LondonRuby for a $150 off • And a few limited 99% discount (grab me after the presentation)
  26. EXTERNAL SYSTEMS • When issuing a command that requires a

    change to an external system • First call the external system (ideally inside the command) • If successful, then record the event
  27. IDEMPOTENCY • Any handler with side-effects should be turned off

    when rebuilding your projections • Emails • Analytics tools (Google Analytics / Mixpanel / Kissmetrics) • In general, any API call to an external system that is done as a result of an event
  28. SCHEMA VERSIONING • You can’t easily change event names •

    You can’t easily change the event schema • Depending on your tolerance you can use JSON / Protocol buffers / AVRO
  29. module EventSourced class Meeting def accept(user_id:) Commands::Meeting::Accept.call(self, user_id: user_id, status:

    @status) end def accepted_handler(event) @attendees_count += 1 end def reschedule(new_time:) Commands::Meeting::Reschedule.call(self, new_time: new_time, status: @status) end def rescheduled_handler(event) params = event.data.symbolize_keys @time = params[:new_time] end def finish Commands::Meeting::Finish.call(self, status: @status) end def finished_handler(event) @status = :finished end def cancel Commands::Meeting::Cancel.call(self, status: @status) end def cancelled_handler(event) @status = :cancelled end end end command command command event handler event handler event handler command event handler
  30. module Commands class Command def self.call(entity, params) new(entity, params).call end

    def initialize(entity, params) @entity = entity @params = params @validator = schema.call(params) end def publish_event(params) Event.create_and_broadcast(params) end def validate @validator.success? ? Response.new(success: true) : Response.new(success: false, errors: @validator.errors) end def schema "#{self.class.name}Schema".constantize end end end
  31. module Commands module Meeting AcceptSchema = Dry::Validation.Schema do required(:user_id).filled required(:status).value(eql?:

    :scheduled) end class Accept < Command def call return validate if validate.failure? publish_event( event_type: 'accepted', entity_type: 'Meeting', entity_id: @entity.id, entity_version: @entity.version, data: { user_id: @params[:user_id] } ) @entity.handle(event) Response.new success: true end end end end