実践nginx〜メルカリの場合〜

1. ࣮ફnginx ʙϝϧΧϦͷ৔߹ʙ Tatsuhiko Kubo@cubicdaiya ALM@2015/04/21
2. ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in Infrastructure Engineering • Mercari, Inc. • Favorites: Go, C, nginx
3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ
4. Agenda ϝϧΧϦͰͷnginxͷ׆༻ࣄྫʹ͍ͭͯ
5. nginx • ੈքͰೋ൪໨ʹར༻͞Ε͍ͯΔOSSͷHTTPαʔό • C10Kʹ଱͑ΒΕΔΞʔΩςΫνϟ • Πϕϯτۦಈ • ϊϯϒϩοΩϯάI/O • ඇಉظI/O • ܰྔͰߴ଎
6. ϝϧΧϦͰΑ͋͘Δޫܠ ࣮ࡍʹ͸͋Δఔ౓ू໿͍ͯ͠·͢
7. Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer
8. Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer
9. ϦόʔεϓϩΩγͱͯ͠ͷnginx • ϦΫΤετͷϩΪϯά • ΞΫηε੍ޚ • ίϯςϯπͷѹॖɾΩϟογϡ • όοϑΝϦϯά • etc…
10. nginx.confͷઃఆྫ server { listen 443 ssl spdy; server_name xxx.yyy; # ϓϩΩγઃఆ proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Connection ""; proxy_http_version 1.1; proxy_buffers 50 8k; # ੩తίϯςϯπ͸nginxͰฦ͢ location ~ /(styles|js|images)/ { root /usr/share/zabbix; expires 30d; } # Zabbix(༻ͷgoogle_auth_proxy)΁ϓϩΩγ location / { proxy_pass http://google_auth_proxy_for_zabbix; } }
11. લஈʹnginxΛஔ͘ϝϦοτ • HTTP or HTTPSαʔόʹඞཁͳλεΫΛҰ௨Γ͜ͳͤΔ • ΞϓϦέʔγϣϯαʔό(e.g. Unicorn)ʹ଍Γͳ͍ػೳΛิ׬ • ωοτϫʔΫϨΠςϯγͷվળ • KeepAlive • gzipѹॖ • TLS Session (Cache | Tickets)ɺOCSP Stapling • SPDY • etc…
12. Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer
13. HTTPS௨৴ͷߴ଎Խ • ҰൠʹHTTPS௨৴Ͱ͸TCP 3-way handshakeʹՃ͑ͯ TLS 3-way handshake͕ൃੜ͢ΔͷͰHTTP௨৴ΑΓ஗͘ͳΔ • HTTPS௨৴ߴ଎ԽͷͨΊͷࡾछͷਆث • TLS Session Cache • TLS Session Tickets • OCSP Stapling
14. TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ • CPUͷϦιʔεͷ࡟ݮ΍ϨΠςϯγͷղফʹޮՌ͕͋Δ
15. TLS Session Cache with nginx
16. TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥΠΞϯ τʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆Ͱηογϣϯ৘ใΛڞ༗Ͱ͖Δ • εϚϗͩͱαϙʔτ͍ͯ͠Δ୺຤͕গͳ͍…
17. TLS Session Tickets with nginx
18. OCSP Stapling • OCSPʹΑΔSSLূ໌ॻͷࣦޮ֬ೝΛαʔόଆͰߦͬ ͯΩϟογϡ • ΫϥΠΞϯτଆͰ΍ΔͱTLSϋϯυγΣΠΫ࣌ʹϨΠ ςϯγ͕ൃੜ͢Δ • ΍ͬͺΓεϚϗͩͱ͋Μ·ΓରԠͯ͠ͳ͍ • Google Chrome for iOSͩͱରԠͯͨ͠
19. OCSP Stapling with nginx ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /etc/nginx/ssl/xxx.yyy.ocsp.crt; resolver xxx.xxx.xxx.xxx valid=30s; resolver_timeout 5s;
20. Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer
21. SPDY with nginx
22. Ͱɺ࣮ࡍͷޮՌ͸ͱݴ͏ͱ
23. TLS Session (Cache|Tickets) ಋೖλΠϛϯά
24. TLS Session (Cache|Tickets) ಋೖλΠϛϯά
25. SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ
26. ΋͏ͪΐͬͱ۩ମతͳྫ
27. ೔ຊ͔Βւ֎ͷZabbix dashboardʹΞΫηε
28. Client • MacBookPro • Google Chrome • HTTP/2༗ޮ • ϒϥ΢βΩϟογϡ͸ৗʹແޮˣ
29. Server(ॳظঢ়ଶ) • Apache(prefork) + mod_php • த਎͸Zabbix • KeepAlive Off • gzipѹॖແޮ • TLS Session Cache & Tickets༗ޮ
30. Server(ॳظঢ়ଶ) "QBDIF
    1PSU HTTPS Server (PPHMF
    "VUI
    1SPYZ "QBDIF
    1PSU SSLऴ୺ Ϣʔβೝূ Zabbix
31. ύϑΥʔϚϯε(ॳظঢ়ଶ) ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT    %BUB
    5SBOTGFS ,#   -PBE
    5JNF TFD   %0.$POUFOU-PBE FE
    5JNF TFD  
32. νϡʔχϯά ͦͷ1 KeepAlive On KeepAliveΛ༗ޮʹ͢Δ
33. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT    %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBE FE
    5JNF TFD TFD TFD
34. νϡʔχϯά ͦͷ2 <IfModule mod_deflate.c> AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE text/js AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/json-rpc </IfModule> gzipѹॖΛ༗ޮʹ͢Δ
35. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT    %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD
36. νϡʔχϯάͦͷ3 "QBDIF
    1PSU HTTPS Server (PPHMF
    "VUI
    1SPYZ "QBDIF
    1PSU SSLऴ୺ Ϣʔβೝূ Zabbix OHJOY
    1PSU SSLऴ୺ΛnginxͰߦ͏
37. νϡʔχϯάͦͷ3 # nginx.conf # in main context worker_processes auto; tcp_nopush on; keepalive_timeout 65s; open_file_cache max=1000 inactive=20s; ssl_session_cache shared:SSL:30m; gzip on; gzip_comp_level 9; gzip_types text/css text/plain text/js text/javascript application/javascript application/json-rpc; # in event context accept_mutex_delay 100ms; # in event context ੩తϑΝΠϧ͸શ෦nginxͰ഑৴͢Δ
38. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT    %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD
39. νϡʔχϯάͦͷ4 listen 443 ssl spdy; SPDY/3.1Λ༗ޮʹ͢Δ
40. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT    %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD ࠷ऴతʹWebϖʔδͷϩʔυ͕࣌ؒ4ඵ͔Β1ඵʹ
41. Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer
42. OHJOY
43. ൚༻ϓογϡ௨஌γεςϜ PS
    CBUDIαʔό
44. ·ͱΊ • nginx͸ • ϝϧΧϦ಺ͰαʔϏεɺࣾ಺γεςϜͰ͍ΖΜ ͳՕॴʹڬΜͰ׆༻͍ͯ͠·͢ • L7ϩʔυόϥϯαʔɺϦόϓϩɺSSLऴ୺αʔ όͱͯ͠ͱͯ΋༏ल • ࠓޙ΋ར༻Օॴ͕૿͑Δ༧ఆ