The Technology behind pixiv Infrastructure

pixivのインフラを支える技術2013
at Python Developers Festa 2013.11

1. harukasan / ͸Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
2. Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪ
   ढ़հ 2003 Kurume National College of Tech.
 - NHK ROBOCON
 - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ.
 - Computational Vision Science 2012 pixiv Inc.
 - Infrastructure team
4. ͜͜ʹpixivͷઆ໌͕ೖΔ
5. Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7 Billion
6. Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
7. ISP Backbone Office IDCF DC New DC 100M 1G 10G 1G 1G line 1G pixiv Network
8. ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
9. http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
10. @cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
11. Agenda pixiv Image Cluster Log Analysis Basis Management Tools
12. pixiv Image Cluster QJYJWͷը૾഑৴Ϋϥελʹ͍ͭͯ
13. pixiv Image Cluster • 2010೥͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ
 ௒ߴ଎ʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
14. Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
15. Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ੒ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ
 εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ཈͑ͭͭ
 Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
16. Cache strategy • ϝϞϦՁ֨ͷ௿Լ • SSDͷՁ͕֨๫མ • ߴIOPSͷSSD͕ొ৔ 2011 ¥40,000
 2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2͸ެশ஋ Intel 910ɺ SSD RAID0ʹ͍ͭͯ͸fioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
17. Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
18. Aggregate image domains • ը૾αʔό͸ϢʔβʔIDϕʔεͰ෼ࢄ
 img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
 ‎Ոఉ༻ϧʔλ͸DNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
19. New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
20. New image store • ࡞੒೔+IDϕʔεͷγʔέϯγϟϧͳURI • 1౓ॻ͖ࠐΜͩϑΝΠϧ͸Read Only • ࠶౤ߘॲཧ͸࡞੒೔Λߋ৽ • ngx_lua/OpenRestyΛ༻͍ͨ࿦ཧ࡟আ
21. Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png 403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
22. Logical Delete local memcached = require "resty.memcached" local uri = ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
23. Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
24. Log Analysis Basis PHP Application MySQL/neoagent Front server - Error Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
25. Error Viewer
26. Slow Query Viewer
27. Kibana 3
28. Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢Δ࢓૊Έ • ϓϩηε͕େྔʹىಈͯ͠΋εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type, $data) # JSON Fluentd in_tail
29. Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer type͸file/࠶ىಈͯ͠΋ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζ͸খ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
30. Management tools ؅ཧܥɺ؂ࢹܥπʔϧʹ͍ͭͯ
31. Monitoring servers/services • ͘͘͝͝Ұൠతͳ؂ࢹϓϩμΫτΛ࢖͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏε؂ࢹ ϗετ಺ϓϩηε؂ࢹ εΫϦϓτ Munin Nagios Monit Cron
32. Cluster Admin ϋʔυ΢ΣΞ৘ใ ϗετͷ༻్ ؂ࢹঢ়ଶ
33. Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ
 ؅ཧԼͷσΟϨΫτϦʹ • ઃఆ൓ө͸capistranoΛ࢖༻ͯ͠શ୆ʹσϓϩΠ • ϗετҰཡ͸APIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update $cap dns:check $cap dns:reload ex: update DNS Record
34. Management Tools • LVS؅ཧը໘ • MySQLͷ஗Ԇ؂ࢹ • αʔϏεͷϦιʔεϞχλϦϯά
35. Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯ঺հ • ࢖͍΍͍͢πʔϧΛࣗ෼͕ͨͪ࢖͍΍͍͢Α͏ʹ • ઑͬͨ͜ͱ͸ͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ
 ΋͍ͬͯ͘