Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load balancer management with Consul

Load balancer management with Consul

Tatsuhiko Kubo

August 02, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Load balancers in • nginx • L7 (HTTP or HTTPS)

    • L4 (TCP) • OpenResty • Dynamic behavior with Lua
  2. nginx nginx nginx ©2011 Amazon Web Services LLC or its

    affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 SPDY/HTTP2 SPDY/HTTP2 EC2 EC2 Load balancer for App
  3. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  4. Load balancer for Search app app app OpenResty Latest Indices

    All Indices Contents cache & Dynamic Balancing HTTP HTTP
  5. in • Service discovery • Load balancer, SMTP, etc… •

    Configuration deployment • TLS Session Tickets, IP black list • Distributed lock • Ensure running only 1 process always in clusters
  6. Service discovery with • Use case in • Internal DNS

    as • Endpoint of internal API, DNS-RR • Via HTTP APIs • Listing nodes in service
  7. Service discovery with 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent

    { “service”: { “name”: “api-internal”, “tags”: [“production”], … } } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  8. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  9. DNS-RR in api-internal service $ dig production.api-internal.service.consul | egrep ‘^production’

    production.api-internal.service.consul. 0 IN A 10.0.1.6 production.api-internal.service.consul. 0 IN A 10.0.1.4 production.api-internal.service.consul. 0 IN A 10.0.1.1 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.5 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.1 $
  10. Endpoint of internal API requests • There are subsystems in

    • Admin tool, Batch, Worker, Web, … • Each subsystem calls Mercari API via HTTP • e.g. production.api-internal.service.consul
  11. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  12. Load balancer for internal API requests private network Multimedia Corporate

    data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP production.api-internal.service.consul production.api-internal.service.consul global network
  13. Listing nodes in service • Use case in • Gathering

    load balancer nodes in application deployment
  14. ChatOps with Slack yes EFQMPZCPU ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  15. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down Mercari deployment App App App nginx nginx nginx
  16. ChatOps with Slack yes EFQMPZCPU rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  17. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up Mercari deployment App App App nginx nginx nginx
  18. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down or up Turn of App App App nginx nginx nginx
  19. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  20. Configuration deployment with • Via • consul event • consul-template

    • Stretcher • Now uses consul event & Stretcher
  21. Load balancer configuration with • Dynamic upstream with consul-template •

    Instead uses ngx_dynamic_upstream • External files • TLS Session Tickets • IP black list
  22. TLS Session Tickets # TLS Session Tickets # openssl rand

    48 > ssl_session_ticket ssl_session_tickets on; ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket; • Client side session cache for TLS • nginx loads from local file on startup • Periodic update is required for forward secrecy • updates automatically with consul event
  23. Configuration 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “watches”:

    [ { “type”: “event”, “name”: “tls-session-ticket-refresh”, “handler”: “/path/to/tls_session_ticket_refresh” ] } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  24. Firing event with consul event $ consul event \ -name=“tls-session-ticket-refresh”

    \ $(openssl rand 48 | base64) consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent Consul server # real processing is # more complicated cat $body | jq -r ‘.Payload’ | \ base64 -d | base64 -d > \ /path/to/tls_session_ticket service nginx reload tls_session_ticket_refresh Automated job Transfer payload
  25. Stretcher • A deployment tool with Consul / Serf event

    • github.com/fujiwara/strecher • Payload is expressed as file-path or URL $ consul event \ -name deploy-xxx \ “s3://example.com/distribution.tar.gz” { “watches”: [ { “type”: “event”, “name”: “deploy-xxx”, “handler”: “/path/to/stretcher” ] } ▪ Trigger ▪ Configuration $ consul event \ -name deploy-xxx \ “/path/to/distribution.tar.gz” or
  26. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System

    Engineer • https://www.mercari.com/jp/jobs/backend/