Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load balancer management with Consul

Load balancer management with Consul

Avatar for Tatsuhiko Kubo

Tatsuhiko Kubo

August 02, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Load balancers in • nginx • L7 (HTTP or HTTPS)

    • L4 (TCP) • OpenResty • Dynamic behavior with Lua
  2. nginx nginx nginx ©2011 Amazon Web Services LLC or its

    affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 SPDY/HTTP2 SPDY/HTTP2 EC2 EC2 Load balancer for App
  3. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  4. Load balancer for Search app app app OpenResty Latest Indices

    All Indices Contents cache & Dynamic Balancing HTTP HTTP
  5. in • Service discovery • Load balancer, SMTP, etc… •

    Configuration deployment • TLS Session Tickets, IP black list • Distributed lock • Ensure running only 1 process always in clusters
  6. Service discovery with • Use case in • Internal DNS

    as • Endpoint of internal API, DNS-RR • Via HTTP APIs • Listing nodes in service
  7. Service discovery with 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent

    { “service”: { “name”: “api-internal”, “tags”: [“production”], … } } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  8. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  9. DNS-RR in api-internal service $ dig production.api-internal.service.consul | egrep ‘^production’

    production.api-internal.service.consul. 0 IN A 10.0.1.6 production.api-internal.service.consul. 0 IN A 10.0.1.4 production.api-internal.service.consul. 0 IN A 10.0.1.1 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.5 $ dig production.api-internal.service.consul | egrep ‘^production’ production.api-internal.service.consul. 0 IN A 10.0.1.3 production.api-internal.service.consul. 0 IN A 10.0.1.2 production.api-internal.service.consul. 0 IN A 10.0.1.1 $
  10. Endpoint of internal API requests • There are subsystems in

    • Admin tool, Batch, Worker, Web, … • Each subsystem calls Mercari API via HTTP • e.g. production.api-internal.service.consul
  11. Load balancer for internal API requests global network private network

    Multimedia Corporate data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP
  12. Load balancer for internal API requests private network Multimedia Corporate

    data center Traditional server Mobile Client Example: Requester Workers SPDY/HTTP2 mercari API HTTP subsystems subsystems HTTP HTTP production.api-internal.service.consul production.api-internal.service.consul global network
  13. Listing nodes in service • Use case in • Gathering

    load balancer nodes in application deployment
  14. ChatOps with Slack yes EFQMPZCPU ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  15. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down Mercari deployment App App App nginx nginx nginx
  16. ChatOps with Slack yes EFQMPZCPU rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh

    mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream Mercari deployment App App App nginx nginx nginx
  17. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up Mercari deployment App App App nginx nginx nginx
  18. ChatOps with Slack yes EFQMPZCPU ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl

    down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down or up Turn of App App App nginx nginx nginx
  19. Listing all nodes in api-internal service $ curl -s \

    consul-server:8500/v1/catalog/service/api-internal \ jq ‘.[].Address’ “10.0.1.1” “10.0.1.2” “10.0.1.3” “10.0.1.4” “10.0.1.5” “10.0.1.6”
  20. Configuration deployment with • Via • consul event • consul-template

    • Stretcher • Now uses consul event & Stretcher
  21. Load balancer configuration with • Dynamic upstream with consul-template •

    Instead uses ngx_dynamic_upstream • External files • TLS Session Tickets • IP black list
  22. TLS Session Tickets # TLS Session Tickets # openssl rand

    48 > ssl_session_ticket ssl_session_tickets on; ssl_session_ticket_key /etc/nginx/ssl/ssl_session_ticket; • Client side session cache for TLS • nginx loads from local file on startup • Periodic update is required for forward secrecy • updates automatically with consul event
  23. Configuration 10.0.1.1 consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent { “watches”:

    [ { “type”: “event”, “name”: “tls-session-ticket-refresh”, “handler”: “/path/to/tls_session_ticket_refresh” ] } 10.0.1.2 10.0.1.3 10.0.1.4 10.0.1.5 10.0.1.6
  24. Firing event with consul event $ consul event \ -name=“tls-session-ticket-refresh”

    \ $(openssl rand 48 | base64) consul-agent consul-agent consul-agent consul-agent consul-agent consul-agent Consul server # real processing is # more complicated cat $body | jq -r ‘.Payload’ | \ base64 -d | base64 -d > \ /path/to/tls_session_ticket service nginx reload tls_session_ticket_refresh Automated job Transfer payload
  25. Stretcher • A deployment tool with Consul / Serf event

    • github.com/fujiwara/strecher • Payload is expressed as file-path or URL $ consul event \ -name deploy-xxx \ “s3://example.com/distribution.tar.gz” { “watches”: [ { “type”: “event”, “name”: “deploy-xxx”, “handler”: “/path/to/stretcher” ] } ▪ Trigger ▪ Configuration $ consul event \ -name deploy-xxx \ “/path/to/distribution.tar.gz” or
  26. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System

    Engineer • https://www.mercari.com/jp/jobs/backend/