Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Scalable Infrastructure at Mercari

Building Scalable Infrastructure at Mercari

Tatsuhiko Kubo

June 24, 2016
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

    ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, widebullet, etc…
  2. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or

    its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
  3. Scale @ • API requests (HTTPS) • 1,300,000 req /

    min • CDN Traffic • 10Gbps • at peak…
  4. Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or

    its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP US nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 RDS SPDY/HTTP2 SPDY/HTTP2 EC2
  5. nginx • HTTP server for general purpose • event-driven, non-blocking

    architecture • 100+ nginx works @ • HTTP/2, SPDY Termination • Reverse Proxy, Load Balancer, …
  6. Why we use nginx? • Scalable • serve over 10K

    connections always per 1 instance • Flexible • TLS Termination, Reverse Proxy, Load Balancing(L4 / L7), Content Caching, and so on… • Extensible • developer can extend nginx with C, Lua, JavaScript, mruby, … • selects C and Lua
  7. Why we use Go? • Simple • build and deployment

    • Enough • scalability, robustness, productivity • Powerful • concurrency support with goroutine and channel
  8. mercari.go • nginx-build - seamless nginx builder (github.com/cubicdaiya/nginx-build) • Gaurun

    - push notification server (github.com/mercari/gaurun) • Widebullet - API gateway (github.com/mercari/widebullet) • etc… ( e.g. CLI in-house operation tools)
  9. Building scalable infrastructure with nginx and Go • has been

    building scalable infrastructure with nginx and Go • Gaurun - Push infrastructure • Pascal - Log analysis infrastructure • Widebullet - API gateway • etc… (Some infrastructure is powered by gRPC)
  10. Gaurun - Push infrastructure ($. "1/T (BVSVO (BVSVO (BVSVO /(*/9

    "1*4FSWFS "1*4FSWFS "1*4FSWFS #BUDI4FSWFS QVTI XJUI LFFQBMJWF )551 1045QVTI
  11. Gaurun - Push infrastructure • The events in are notified

    to users. • Purchase, Like, Comment, Shipment, … • Push notification to all users at once? • Over a few tens of millions pushʂ • High performance and scalability are required for pushing quickly
  12. Gaurun • Push notification server in Go • github.com/mercari/gaurun •

    Provides HTTP APIs for push and metrics • Supports GCM and APNs ( HTTP/2, too) • Scalable • In , a few Gaurun instances process all pushes.
  13. 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Scientist Analyze by SQL

    send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Log analysis infrastructure in_tail & out_forward
  14. PascalʙLog analysis infrastructureʙ • Built with the software blocks below

    • Puree, OpenResty (ngx_lua), Fluentd, fluent-agent-hydra, Google BigQuery • Aggregate various logs to Google BigQuery • Event in app • A/B Testing • etc…
  15. • Low resource usage is required • Pascal indicates modestly

    high workload • OpenResty processes a lot of JSONs and outputs various logs • Requests are come from not only device but internal subsystems (e.g. API). • OpenResty and fluent-agent-hydra are very fast and efficient PascalʙLog analysis infrastructureʙ
  16. Widebullet - API gateway • In , subsystems have been

    increased day by day. • some subsystem must call multiple APIs at once • Call APIs concurrently for high performance • Widebullet proxies multiple API calls at once
  17. Summary • has been building scalable infrastructure with nginx and

    Go • Why we use nginx and Go? • nginx • Scalable, Flexible, Extensible • Go • Simple, Enough, Powerful • These infrastructures are constructed by SREs and Backend System Engineers
  18. We are hiring! • SRE • https://www.mercari.com/jp/jobs/sre/ • Backend System

    engineer • https://www.mercari.com/jp/jobs/backend/