Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Thalassa - DenverJS

Mike Brevoort
September 12, 2013

Thalassa - DenverJS

Thalassa is a system of components geared to enable continuous deployment scenarios through dynamic configuration of HAProxy load balancers.

http://www.meetup.com/Denver-JS/events/129876632/

Mike Brevoort

September 12, 2013
Tweet

More Decks by Mike Brevoort

Other Decks in Technology

Transcript

  1. • no downtime • no user impact • no connection

    drops • repeatable • reliable Continuos Deployment Friday, September 13, 13
  2. A/B style deployment • Minimize disruption • Reliable fast rollbacks

    • Deployments don’t cripple capacity Friday, September 13, 13
  3. The Problem How do we automate the load balancer and

    A/B style deployments? for heterogeneous apps? (node.js, Java, etc.) in heterogenous environments (public/private cloud, bare metal) Friday, September 13, 13
  4. Could we use Seaport and a node as a dynamic,

    programmable proxy? the “itch” https://github.com/substack/seaport Friday, September 13, 13
  5. Why node.js? • Many connections • Streaming data • Relatively

    lightweight • Very programmable • Seaport was an excellent head-start Friday, September 13, 13
  6. HAProxy • Software router/load balancer • A reliable workhorse •

    Feature rich • Can dynamically reload configuration without dropping connections Friday, September 13, 13
  7. Thalassa • Registry of services • API for configuring HAProxy

    • Dynamically set HAProxy backend servers based on registry services name and version • Fault tolerant, do no harm fka “Spindrift” Friday, September 13, 13
  8. Thalassa • Lightweight service registry • Formerly build on top

    of @substack’s Seaport • Ping and expire based approach through HTTP API • Built on Redis, axon for pub-sub • https://github.com/PearsonEducation/ thalassa Friday, September 13, 13
  9. Aqueduct • REST API for HAProxy • Manage HAProxy process

    • Dynamically generates config • Streams HAProxy stats • Configuration as a CRDT • enables peering of Aqueduct servers • https://github.com/PearsonEducation/ thalassa-aqueduct Friday, September 13, 13
  10. Thalassa Server Aqueduct HAProxy A@1 A@1 A@1 A@1 80 FE

    BE axon socket Friday, September 13, 13
  11. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80

    FE BE axon socket Friday, September 13, 13
  12. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80

    FE BE axon socket Friday, September 13, 13
  13. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 A@1 80

    FE BE streaming stats axon socket Friday, September 13, 13
  14. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 FE

    BE streaming stats axon socket Friday, September 13, 13
  15. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2

    A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
  16. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2

    A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
  17. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2

    A@2 A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
  18. Thalassa Server Aqueduct HAProxy Crowsnest A@1 A@1 A@1 80 A@2

    A@2 A@2 FE BE streaming stats axon socket Friday, September 13, 13
  19. Thalassa Registry • Redis Implementation • Key/Value for each registration

    • Sorted set of keys ordered by time to expire Friday, September 13, 13
  20. local res = redis.call('ZRANGEBYSCORE','__thalassa.registrations', 0, 1, 'LIMIT', 0, 100 )

    if #res > 0 then redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 ) return res else return false end function onOnline (reg) { pub.send(reg.id, 'online', reg.stringify()); } function onOffline(regId) { pub.send(regId, 'offline'); } The Reaper Friday, September 13, 13
  21. Aqueduct Config Templates • haproxy.cfg.tmpl template for haproxy.cfg • Handlebars.js

    • Handlebars macros https://github.com/PearsonEducation/thalassa-aqueduct/blob/master/default.haproxycfg.tmpl Friday, September 13, 13
  22. global log 127.0.0.1 local0 log 127.0.0.1 local1 notice daemon maxconn

    4096 user haproxy group haproxy # stats socket {{haproxySocketPath}} user USER_RUNNING_NODE_PROCESS level admin stats socket {{haproxySocketPath}} level admin defaults log global option dontlognull option redispatch retries 3 maxconn 2000 timeout connect 5000ms timeout client 50000ms timeout server 50000ms listen stats :1988 mode http stats enable stats uri / stats refresh 2s stats realm Haproxy\ Stats stats auth showme:showme {{#each frontends}} frontend {{name}} {{frontendHelper this}} {{/each}} {{#each backends}} backend {{name}} {{backendHelper this}} {{/each}} Friday, September 13, 13
  23. Use of Streams • Aqueduct streams config and HAProxy stats

    to Crowsnest over a websocket • Crowsnest aggregates and streams to the browser • MuxDemux Friday, September 13, 13
  24. registrations config stats stats config stats Aqueduct Aqueduct config Crowsnest

    Server Aqueduct Thalassa Server control stream MuxDemux Streams Friday, September 13, 13
  25. When? • Modules will be officially published this month within

    the next several weeks • Repositories http://github.com/PearsonEducation Friday, September 13, 13