Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Heroku Postgres

Heroku Postgres

Talk delivered at Burlington Ruby Conf, 07/28/2012

Harold Giménez

July 28, 2012
Tweet

More Decks by Harold Giménez

Other Decks in Programming

Transcript

  1. Heroku Postgres The Tale of Conceiving and Building a Leading

    Cloud Database Service 1 Harold Giménez @hgimenez 1 Saturday, July 28, 12
  2. 12 Heroku Postgres v.0.pre.alpha • A sinatra app implementing the

    heroku addons API • create servers • install postgres service • create databases for users - a “Resource” • Sequel talks to postgres • stem talks to AWS 12 Saturday, July 28, 12
  3. Resource 14 { database: ‘d4f9wdf02’, port: 5432, username: ‘uf0wjasdf’, password:

    ‘pf14fhjas’, created_at: ‘2012-05-02’, state: ‘available’ } 14 Saturday, July 28, 12
  4. Server 15 { elastic_ip: ‘192.168.0.1’, instance_id: ‘i-2efjoiads’, ami: ‘pg-prod’, availability_zone:

    ‘us-east-1’, created_at: ‘2012-05-02’, state: ‘booting’ } 15 Saturday, July 28, 12
  5. ...and a thin admin web interface 16 erb templates in

    sinatra endpoint 16 Saturday, July 28, 12
  6. 24 class Resource def feel observations.create( Feeler.new(self).current_environment ) end end

    class Feeler def current_environment { service_available?: service_available?, open_connections: open_connections, row_count: row_count, table_count: table_count, seq_scans: seq_scans, index_scans: index_scans } end end monitoring 24 Saturday, July 28, 12
  7. 25 class Resource include Stateful state :available do unless service_available?

    transition :unavailable end end end resource = Resource.new resource.transition :available resource.feel resource.tick puts resource.state # ‘unavailable’ workflow 25 Saturday, July 28, 12
  8. 26 module Stateful def self.included(base) base.extend ClassMethods end module ClassMethods

    def state(name, &block) states[name] = block end def states; @states ||= {}; end end def tick self.instance_eval( &self.class.states[self.state.to_sym] ) end def transition(state) # log and assign new state end end workflow 26 Saturday, July 28, 12
  9. 29 db1 db2 db3 db4 db5 db6 db7 db8 db9

    ... dbn db1.feel db1.tick 29 Saturday, July 28, 12
  10. 30 db2 db3 db4 db5 db6 db7 db8 db9 ...

    dbn db1 db2.feel db2.tick enqueue(db1) 30 Saturday, July 28, 12
  11. Continuous Protection 37 • Write-Ahead Log segments shipped to durable

    storage every 60 seconds • We can replay these logs on a new server to recover your data • https://github.com/heroku/WAL-E 37 Saturday, July 28, 12
  12. 62 post “/work” do puts “starting to do work” worker

    = Worker.new(params) begin worker.lift_things_up worker.put_them_down rescue WorkerError => e puts “Fail :( #{e.message}” status 500 end puts “done doing work” status 200 end 62 Saturday, July 28, 12
  13. 63 $ heroku logs --tail 2012-07-28T02:43:35 [web.4] starting to do

    work 2012-07-28T02:43:35 [web.4] Fail :( invalid worker, nothing to do 2012-07-28T02:43:35 heroku[router] POST myapp.com/work dyno=web.4 queue=0 wait=0ms service=14ms status=500 bytes=643 63 Saturday, July 28, 12
  14. 65 post “/work” do log(create_work: true, request_id: uuid) do worker

    = Worker.new(params.merge(uuid: uuid)) begin worker.lift_things_up worker.put_them_down rescue WorkerError => e log_exception(e, create_work: true) end end end helpers do def uuid SecureRandom.uuid end end 65 Saturday, July 28, 12
  15. 66 require ‘scrolls’ module App module Logs extend self def

    log(data, &block) Scrolls.log(with_env(data), &block) end def log_exception(exception, data, &block) Scrolls.log_exception(with_env(data), &block) end def with_env(hash) { environment: ENV[‘RACK_ENV’] }.merge(data) end end end 66 Saturday, July 28, 12
  16. 67 $ heroku logs --tail 2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=start

    2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=exception message=invalid worker, nothing to do 2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=finish elapsed=53 2012-07-28T02:43:35 heroku[router] POST myapp.com/work dyno=web.4 queue=0 wait=0ms service=14ms status=500 bytes=643 67 Saturday, July 28, 12
  17. current tooling 74 • still using sequel and sinatra •

    fog displaced stem • backbone.js for web UIs • fernet for auth tokens, valcro for simple validations, QueueClassic for job queues • Wal-e for durability • python, go and bash in some subsystems 74 Saturday, July 28, 12
  18. • managing databases is hard • start simple • extract

    (and share) reusable code • separate concerns into services • use the right tool (framework, library, language) • learn to love your event stream, metrics for everything 75 lessons 75 Saturday, July 28, 12
  19. select a.body from answers a inner join questions q on

    a.id = q.answer_id; 77 77 Saturday, July 28, 12