Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Technology behind pixiv Infrastructure
Search
Harukasan
PRO
November 30, 2013
Technology
10
4k
The Technology behind pixiv Infrastructure
pixivのインフラを支える技術2013
at Python Developers Festa 2013.11
Harukasan
PRO
November 30, 2013
Tweet
Share
More Decks by Harukasan
See All by Harukasan
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
400
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
710
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.4k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.5k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.5k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
2.9k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
440
Other Decks in Technology
See All in Technology
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
300
第3回Snowflake女子会_LT登壇資料(合成データ)_Taro_CCCMK
tarotaro0129
0
200
LINEヤフーのフロントエンド組織・体制の紹介【24年12月】
lycorp_recruit_jp
0
530
あの日俺達が夢見たサーバレスアーキテクチャ/the-serverless-architecture-we-dreamed-of
tomoki10
0
460
PHP ユーザのための OpenTelemetry 入門 / phpcon2024-opentelemetry
shin1x1
1
270
Oracle Cloud Infrastructure:2024年12月度サービス・アップデート
oracle4engineer
PRO
0
200
どちらを使う?GitHub or Azure DevOps Ver. 24H2
kkamegawa
0
840
ずっと昔に Star をつけたはずの思い出せない GitHub リポジトリを見つけたい!
rokuosan
0
150
組織に自動テストを書く文化を根付かせる戦略(2024冬版) / Building Automated Test Culture 2024 Winter Edition
twada
PRO
17
4.6k
生成AIのガバナンスの全体像と現実解
fnifni
1
190
祝!Iceberg祭開幕!re:Invent 2024データレイク関連アップデート10分総ざらい
kniino
3
310
フロントエンド設計にモブ設計を導入してみた / 20241212_cloudsign_TechFrontMeetup
bengo4com
0
1.9k
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
67
10k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.9k
Code Reviewing Like a Champion
maltzj
520
39k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
GraphQLとの向き合い方2022年版
quramy
44
13k
A Philosophy of Restraint
colly
203
16k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
127
18k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
2
170
Embracing the Ebb and Flow
colly
84
4.5k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
6.9k
How to Think Like a Performance Engineer
csswizardry
22
1.2k
The Pragmatic Product Professional
lauravandoore
32
6.3k
Transcript
harukasan / Δ͔͞Μ at #pyfes 2013.11 ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
Harukasan@pixiv a.k.a MICHII Shunsuke ಓҪढ़հ 2003 Kurume National College of
Tech. - NHK ROBOCON - ACM-ICPC 2008 Kyushu Inst. of Technology 2010 Tsukuba Univ. - Computational Vision Science 2012 pixiv Inc. - Infrastructure team
None
͜͜ʹpixivͷઆ໌͕ೖΔ
Server 400+ Traffic 10Gbps+ Team member 6 Monthly PV 3.7
Billion
Bases of pixiv Infrastructure Office IDCF DC Developments Testing Log
Analytics Small Services Main Applications DB Image Cluster New DC Image Cluster
ISP Backbone Office IDCF DC New DC 100M 1G 10G
1G 1G line 1G pixiv Network
ͷΠϯϑϥΛࢧ͑Δٕज़ 2013
http://www.slideshare.net/kamipo/pixiv @kamipo pixivͷΠϯϑϥΛࢧ͑Δٕज़ / ςΫηϛ2009
@cubicdaiya inside pixiv’s infrastructure / PHP Conference 2013 http://www.slideshare.net/cubicdaiya/inside-pixiv-infrastructure
Agenda pixiv Image Cluster Log Analysis Basis Management Tools
pixiv Image Cluster QJYJWͷը૾৴Ϋϥελʹ͍ͭͯ
pixiv Image Cluster • 2010͔Βӡ༻։࢝ • pixivͷϝΠϯίϯςϯπͰ͋ΔΠϥετΛ ߴʹॲཧ͢ΔͨΊʹ࠷దԽ • શτϥϑΟοΫͷ90%Ҏ্Λࡹ͍͍ͯΔ
Image Cluster nginx Front Cache DNS Round-Robin ATS Cache nginx
Dispatch nginx Front Cache nginx Front Cache nginx Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net SmallLight
Cache strategy • ϝϞϦͱσΟεΫͷ2ஈΩϟογϡߏ • τϥϑΟοΫ͕૿͑ΔʹͭΕͯ εΠονؒτϥϑΟοΫ͕ແࢹͰ͖ͳ͘ͳͬͨ • ωοτϫʔΫτϥϑΟοΫΛ͑ͭͭ Ωϟογϡ༰ྔͷ֬อ͢Δඞཁ
Cache strategy • ϝϞϦՁ֨ͷԼ • SSDͷՁ͕֨མ • ߴIOPSͷSSD͕ొ 2011 ¥40,000
2013 ¥20,000 256G READ WRITE PRICE ioDrive2 785GB MLC*1 215,000/230,000 IOPS I don’t know Intel 910 800GB 100,000/ 75,000 IOPS ¥400,000 SSD 256GBx3 RAID0 80,000/ 50,000 IOPS ¥60,000 *1: ioDrive2ެশ Intel 910ɺ SSD RAID0ʹ͍ͭͯfioʹΑΔଌఆ 16G ECC RDIMM ¥20,000
Cache strategy ound-Robin nginx Front Cache nginx Front Cache nginx
Front Cache ATS Cache ATS Cache ATS Cache Consistent Hashing nginx Dispatch nginx Dispatch nginx Dispatch Apache Origin nginx Thumbnail DNS Round-Robin i1.pixiv.net i2.pixiv.net 64GB Memory - nginx cache on tmpfs - cache hit rate: 50% - reduce network traffic 256GB SSD x3 RAID0 - Apache Traffic Server (standalone) - cache hit rate: 80-90% SSD SSD HDD Original & BIG Thumb. Small Thumbnails
Aggregate image domains • ը૾αʔόϢʔβʔIDϕʔεͰࢄ img01.pixiv.net - img1XX.pixiv.net • 1ϖʔδͰ40-60ճDNSϦΫΤετ͕ൃੜ
Ոఉ༻ϧʔλDNSղܾ͕Ͱ͖ͳ͘ͳΔ • શը૾ͷURIΛมߋͯ͠ରԠ OLD: http://img01.pixiv.net/img/****/*****.jpg NEW: http://i1.pixiv.net/img01/img/****/*****.jpg
New Image Store ৽͍͠ը૾ετϨʔδํࣜʹ͍ͭͯ
New image store • ࡞+IDϕʔεͷγʔέϯγϟϧͳURI • 1ॻ͖ࠐΜͩϑΝΠϧRead Only • ࠶ߘॲཧ࡞Λߋ৽
• ngx_lua/OpenRestyΛ༻͍ͨཧআ
Logical Delete ! Kyototycoon ngx_lua / nginx null 404 /img02.png
403 /img03.png 404 /img05.png 404 /img08.png 403 " " GET /img01.png GET /img03.png 404
Logical Delete local memcached = require "resty.memcached" local uri =
ngx.var.request_uri local memc = memcached:new() . . . local val, flags, err = memc:get(request_uri) if val and val ~= "200" then exit(tonumber(val)) end logical_delete.lua location / { access_by_lua_file logical_delete.lua; }
Log Analysis Basis QJYJWͷϩάղੳج൫ʹ͍ͭͯ
Log Analysis Basis PHP Application MySQL/neoagent Front server - Error
Log - Login Log - Activity Log - Slow Query - Access Log MongoDB Elasticsearch Fluentd File System
Error Viewer
Slow Query Viewer
Kibana 3
Output Log • JSONΛॻ͖ग़ͯ͠Fluentd͕tail͢ΔΈ • ϓϩηε͕େྔʹىಈͯ͠εϧʔϓοτ͕མͪͳ͍ PHP Exception Handler Logger::write($type,
$data) # JSON Fluentd in_tail
Fluentd config <source> type tail path /var/tmp/log/activity.log pos_file /var/tmp/fluentd/activity.pos tag
activity format json # JSONܗࣜΛಡΈࠐΉ </source> <match activity> type forward_with_hostname # HostΛೖΕͯforward flush_interval 1s # ඞͣ1ඵ ! buffer_type file # buffer typefile/࠶ىಈͯ͠ফ͑ͳ͍ buffer_path /var/tmp/fluentd/buffer/activity.*.buffer buffer_chunk_limit 2m # chunkαΠζখ͞Ί buffer_queue_limit 128 # Ͳͷ͘Β͍ফ͑ͪΌ͍͚ͳ͍͔ … </match>
Management tools ཧܥɺࢹܥπʔϧʹ͍ͭͯ
Monitoring servers/services • ͘͘͝͝ҰൠతͳࢹϓϩμΫτΛ͍ͬͯΔ Ϧιʔεάϥϑ ϗετ/αʔϏεࢹ ϗετϓϩηεࢹ εΫϦϓτ Munin Nagios
Monit Cron
Cluster Admin ϋʔυΣΞใ ϗετͷ༻్ ࢹঢ়ଶ
Capistrano/Subversion • /etc/ҎԼͷઃఆϑΝΠϧ͕ͦͷ··subversionͷ ཧԼͷσΟϨΫτϦʹ • ઃఆөcapistranoΛ༻ͯ͠શʹσϓϩΠ • ϗετҰཡAPIܦ༝ͰऔಘͰ͖ΔΑ͏ʹ $cap dns:update
$cap dns:check $cap dns:reload ex: update DNS Record
Management Tools • LVSཧը໘ • MySQLͷԆࢹ • αʔϏεͷϦιʔεϞχλϦϯά
Conclusion • pixivΛࢧ͑Δج൫γεςϜʹ͍ͭͯհ • ͍͍͢πʔϧΛ͕͍͍ࣗͨͪ͢Α͏ʹ • ઑͬͨ͜ͱͤͣɺແཧͤͣӡ༻Ͱ͖Δঢ়ଶʹ ͍ͬͯ͘