Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
CoreOSで運用するために考えないといけないこと / CoreOS in pixiv
Search
Harukasan
PRO
April 09, 2015
Technology
23
7.8k
CoreOSで運用するために考えないといけないこと / CoreOS in pixiv
CoreOS Meetup Tokyo #1
2015-04-09
Harukasan
PRO
April 09, 2015
Tweet
Share
More Decks by Harukasan
See All by Harukasan
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
320
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
470
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
790
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.6k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
harukasan
PRO
5
1.6k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
3k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3.1k
Other Decks in Technology
See All in Technology
Azure & DevSecOps
kkamegawa
2
200
Microsoft Fabric のライセンスについて
ryomaru0825
2
3.4k
Google Cloud Next 2025 Recap 生成AIモデルとマーケティングでのコンテンツ生成 / Generative AI models and content creation in marketing
kyou3
0
350
RubyKaigi NOC 近況 2025
sorah
3
1.2k
技術選定の審美眼(2025年版) / Understanding the Spiral of Technologies 2025 edition
twada
PRO
64
16k
Amplifyとゼロからはじめた AIコーディング。失敗と気づき
mkdev10
1
160
SaaS公式MCPサーバーをリリースして得た学び
kawamataryo
5
1.4k
Part1 GitHubってなんだろう?その2
tomokusaba
2
830
PythonツールであるpygnmiをSONiCのgNMIに対して使ってみた
sonic
0
220
Sleep-time Compute: LLM推論コスト削減のための事前推論
sergicalsix
1
150
DynamoDB のデータを QuickSight で可視化する際につまづいたこと/stumbling-blocks-when-visualising-dynamodb-with-quicksight
emiki
0
180
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
720
Featured
See All Featured
How to Think Like a Performance Engineer
csswizardry
23
1.6k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.5k
GraphQLとの向き合い方2022年版
quramy
46
14k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Testing 201, or: Great Expectations
jmmastey
42
7.5k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
41
2.3k
Docker and Python
trallard
44
3.4k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Bash Introduction
62gerente
613
210k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.7k
Site-Speed That Sticks
csswizardry
6
560
It's Worth the Effort
3n
184
28k
Transcript
CoreOSͰӡ༻͢ΔͨΊʹ ߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ Harukasan / MICHII Shunsuke CoreOS Meetup Tokyo #1
2015-04-09
Harukasan / MICHII Shunsuke • 2012ʹ৽ଔͱͯ͠ΠϯϑϥνʔϜʹଐ • ίϯςϯπ৴Λ͡Ίͱͯ͠αʔϏεશମͷج൫Λ୲ • 16Gbps͘Β͍ͷը૾৴
• αʔϏεϑϩϯτ(nginx) • Fluentd / Kibana / BigQuery • ٕज़ܥࡶࢽͷࣥචͳͲ
beer@harukasan http://harukasan.jp/beers • ौ୩ͰϏʔϧΛҿΉΠϕϯτΛ͍ͬͯ·͢ • ࠓ݄ͷΠϕϯτ໌༻ҙ͠·͢ʢͨͿΜ23ʁʣ
͡Ίʹ • pixivʹ͓͚ΔCoreOS • ͳʹ͕͍ͨ͠ͷ͔ • CoreOSΛ͏ͱԿΛߟ͑ͳ͍ͱ͍͚ͳ͍ͷ͔ • ͜͜1ϲ݄͘Β͍΄ͱΜͲ৮ͬͯͳ͍……
CoreOS in pixiv
CoreOS in pixiv • Relase 554͔Β͍࢝Ίͨ • Ұ෦ͷϓϩμΫγϣϯڥͰ༻ • IDCFΫϥυ্ʹߏங
• ࠷খߏ • ·͍ͩΖ͍Ζͱࢼ͍ͯ͠Δͱ͜Ζ
• pixivͷອը࡞͔ΒΦεεϝ࡞Λ ӾཡɺݕࡧͰ͖ΔAndroid͚ΞϓϦ • 20152݄4ϦϦʔε • Build with Play framework
/ Scala ΦεεϝϚϯΨΛ΄΅ຖߋ৽ʂ pixivϚϯΨ
App App App App LB RPC RPC RPC RPC LB
pixiv DBs manga-app DB CoreOS CoreOS CoreOS CoreOS ΦϯϓϨϛε IDCF Cloud Request flow Manage Manage L3 Private connect
App App App App LB RPC RPC RPC RPC LB
Manage pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS fleetctl Jenkins pull pploy Deployment flow ΦϯϓϨϛε IDCF Cloud L3 Private connect
App App App App LB RPC RPC RPC RPC LB
pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS Monitoring flow ΦϯϓϨϛε IDCF Cloud L3 Private connect td-agent td-agent td-agent td-agent dd-agent dd-agent dd-agent dd-agent Fluentd Logs Munin Nagios
ਓͷԹ͔ΈΛײ͡ΔσϓϩΠ • cloud-configͰઃఆ͕ྲྀ͠ࠐ·ΕΔ • σϓϩΠ৬ਓʹΑΔϩʔϦϯάσϓϩΠ • fleetctlͰϦελʔτ͢Δ͜ͱͰσϓϩΠ͢Δ • ࣗಈԽ͍ͨ͠ export
FLEETCTL_ENDPOINT=http://app-‐1:4001/ NUM=1 fleetctl stop app-‐a@${NUM}.service fleetctl start app-‐a@${NUM}.service
ͳͥCoreOSͳͷ͔
ͳͥCoreOSΛ࠾༻ͨ͠ͷ͔ • Play framework / ScalaͷΞϓϦέʔγϣϯͷґଘ ཧͱ͔ཧ͢Δͷ͕໘ͩͬͨͷͰίϯςφʹด͡ ࠐΊ͔ͨͬͨ • ίϯςφ͔͠ಈ͔͞ͳ͍ϗετΛӡ༻͢Δͷ͕໘
• ίϯςφΛಈ͔͚ͩͩͬͨ͢ΒCoreOSͰྑ͍ͷͰ
ͳʹ͕͍ͨ͠ͷ͔ • ΠϯϑϥνʔϜͱࣗΒͷࣄΛͳ͘͠ଓ͚Δ͜ͱ ͕৬ͱͯ͠ཁٻ͞ΕΔࣗݾໃ६ͨ͠ଘࡏͰ͋Δ • ͨͩ͠ؾΛൈ͘ͱࣄ͕૿͑ଓ͚ͯࢮ͵ • ΊΜͲ͍͘͜͞ͱͰ͖Δ͚ͩΓͨ͘ͳ͍
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • όʔδϣϯͷࠩҟ • ੬ऑੑ • ґଘؔ • Ϣʔβੜσʔλ •
etc.
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • Immutable • ίϯςφҎ֎ͷঢ়ଶΛม͑ͳ͍ • /usrҎԼͦͦॻ͖͑Ͱ͖ͳ͍ • Disposable •
ίϯςφΛ͍ͭͰࣺͯΔ͜ͱ͕Ͱ͖Δ • ͳΜ͔͓͔͘͠ͳͬͨΒࣺͯΕྑ͍
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏεཧͯ͢systemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •
fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏεཧͯ͢systemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •
fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ • ͜ΕҎ্ඞཁͳΒKubernetes͕ඞཁʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • ͭʹSSHͰ͖ͳ͍༁Ͱͳ͍ • ύοέʔδཧγεςϜ͍Βͳ͍ • ͍͍ͩͨͷ͜ͱsystemd͕Δ • ։ൃ࣌ʹDockerίϯςφΛಈ͔͢Πϯελϯε ͱͯ͠ศར
Container Deployment • όΠφϦʹؔ࿈͢Δͯ͢ͷґଘؔΛίϯςφͷத ʹด͡ࠐΊΔ • ίϯςφͷ֎ͷঢ়ଶؾʹ͢Δඞཁ͕ͳ͍ /etc/ *****.conf …
/usr/local/bin/ ***** … /usr/lib/ **** … /var/ **** … Container
Container Deployment • ґଘؔΛ։ൃऀ͕ཧͰ͖Δ • ԾΠϯελϯεΈ͍ͨʹͨ͘͞ΜͷϊʔυΛ1ͭͷ ཧϚγϯʹಉډͤ͞Δͷͱͪΐͬͱҧ͏ • ͋͘·ͰґଘؔΛύοέʔδϯάͨ͠1ϓϩηε ͱࢥͬͨΒΘ͔Γ͍͢
Web Application in a Container • ΞϓϦέʔγϣϯϓϩηείϯςφʹ͍͍ͯΔ • 1ϓϩηεʢجຊతʹʣ •
ґଘ͢ΔϥΠϒϥϦ͕ଟ͍ • ϝϞϦҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ • LBͰϦΫΤετΛεΠονͰ͖Δ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ • CoreOSͱ͍͏͔ɺͲ͏ͬͯ֎෦ʹґଘ͠ͳ͍ঢ়ଶ Λͭ͘Δ͔ • ·ͩ͏·͍ํ๏͕ݟ͔ͭͬͯͳ͍͜ͱ͕ଟ͍ • γεςϜΦʔέετϨʔγϣϯ •
σϓϩΠϝϯτ • ϞχλϦϯά • োରԠ • ࣗಈΞοϓάϨʔυ
etcdʹ͓͚Δ࠷খߏ • ΫϥελΛ࡞Δͱ͖࠷Ͱ4ʹ͓ͯ͘͠ • 3ͩͱ1མ͚ͪͨͩͰεϓϦοτϒϨΠϯ
͏1ඞཁ
γεςϜΦʔέετϨʔγϣϯ • fleetͰͰ͖Δͷجຊతʹʮ͜ͷsystemdαʔϏεΛ ΫϥελͰ͍ͭ͘ಈ͔͔͢ʯͱ͍͏͜ͱ͚ͩ • αʔϏεϝϯόͱ͔Ϛελϊʔυͱ͔Λཧ͢Δػೳ ͳ͍ • ͪΖΜΦʔτεέʔϦϯάͱ͔ͳ͍ •
etcdΛͬͯࣗͰ͕ΜΔʁ • KubernetesͰΔʁ • LB͘Β͍͏ͪΐͬͱ؆୯ʹΓ͍ͨ
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά • ίϯςφͷstart/stop࣌ʹLBͷAPIΛୟ͘ • systemdͷαʔϏεϑΝΠϧʹίϚϯυΛՃ͢Δ https://github.com/coreos/elb-presence • LBͷAPIΛୟ͚ͩ͘ͳͷͰൺֱత؆୯ •
ϋʔτϏʔτνΣοΫͰ֎ΕΔ͚ͩͰྑ͍
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά • etcdΩʔ͕มߋ͞Εͨͱ͖ʹϑοΫͰ͖Δ • ϑοΫͯ͠LBͷઃఆΛม͑Εྑͦ͞͏ • vulcand https://github.com/mailgun/vulcand •
confd + nginx https://github.com/kelseyhightower/confd
σϓϩΠϝϯτ • fleetͩͱϩʔϦϯάϦελʔτͰ͖ͳ͍ • ϩʔϦϯάϦελʔτ͢ΔΑ͏ͳΈΛͭ͘Δʹ ࣗͰεΫϦϓτΛॻ͔͘͠ͳ͍ • ଞͷίϯςφ͕ಈ͍͍ͯΔϊʔυͰಈ͔ͳ͍Α͏ʹ ࢦఆ
[Unit] Description=app After=docker.service Requires=docker.service [Service]
Restart=always TimeoutStartSec=0 ExecStartPre=-‐/usr/bin/docker kill app-‐%i ExecStartPre=-‐/usr/bin/docker rm app-‐%i ExecStartPre=/usr/bin/docker pull … ExecStart=/usr/bin/docker run —name app-‐%i … ExecStop=/usr/bin/docker stop app-‐%i [X-‐Fleet] Conflicts=app@*.service
[email protected]
:
ϞχλϦϯά • ίϯςφ͝ͱʹϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ • ίϯςφ໊Λ͚ͭͣʹMackerelͰࢹͯͨ͠Β σϓϩΠͷʹࢹର͕૿͔͑ͯͬͨ • DataDogͰͲ͏ʹ͔ͳͬͯΔ
ϞχλϦϯά • ͳʹΛϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ͷ͔ • αʔϏεϨϕϧ • ϊʔυϨϕϧ • ίϯςφϨϕϧ •
ͳΜͰݟͨ͘ͳΔ͚Ͳຊʹશ෦ݟͳ͍ͱ ͍͚ͳ͍ͷ͔ߟ͑Δඞཁ͕͋Δ • 1ϊʔυ=1ίϯςφͰಈ͔͢ͷͰ͋Εࠓ·ͰͲ͓Γ ͷϞχλϦϯάͱͦΜͳʹมΘΒͳ͍
ϩάసૹ • ֤ϊʔυʹtd-agentͷίϯςφΛཱ͍ͯͯΔ • cloud-configʹॻ͍ͯىಈͯ͠ྑͦ͞͏
োରԠ • ࠓͷͱ͜Ζϗετʹґଘ͢Δোʹૺ۰ͯ͠ͳ͍ • ϋʔτϏʔτ͕ಧ͍ͯΔঢ়گͩͱLB͔ΒखಈͰ ֎ͨ͠Γ͠ͳ͍ͱ͍͚ͳ͍͔ʁ
ࣗಈΞοϓάϨʔυ • ϊʔυ͕མͪΔ͚ͩͱࢥ͑ϊʔυোͱมΘΒͳ͍ • ϩοΫΛ͔͚ͳ͕Βͬͯ͘ΕΔͷͰϩʔϦϯάΞο ϓάϨʔυ͞ΕΔʢฒྻࢦఆͰ͖Δʣ • fleetͷόʔδϣϯ্͕͕ͬͨͱ͖ʹཧϗετଆͷ fleetctl͕ߋ৽͞Εͯͳࣦͯ͘ഊͨ͜͠ͱ͕͋ͬͨ
ࣗಈΞοϓάϨʔυ • ৽͍ͭ͠Λ͍͍͚ͨͲAlphaόάͬͯΔͷͰ Ͳ͏͠Α͏ • ࣗͰϏϧυͨ͠Βҙຯͳ͍͠
ωοτϫʔΫ • ಛʹωοτϫʔΫΛΔϞνϕʔγϣϯͳ͍ • ϙʔτϑΥϫʔσΟϯάͰͦΕ΄Ͳͳ͍ • ϗετωοτϫʔΫͰྑ͍
ηΩϡϦςΟ • CoreOS͕໘Έͯ͘ΕΔͷϗετOS͚ͩ • ίϯςφͷ੬ऑੑؔͳ͍ • Dockerͩͱίϯςφʹόʔδϣϯ͕ͳ͍ͷͰ͍ͬͣ͜ ͭ֬ೝ͠ͳ͍ͱ͍͚ͳ͍
·ͱΊ • ݁ہͳΜ͔ΜඞཁʹͳΔ • ͦΕKubernetesͰ(ུ • ͦΕGCPͰ(ུ • ίϯςφ͡Όͳͯ͘AMIͰ… •
Ͳ͜·ͰΔඞཁ͕͋Δ͔ߟ͑Δඞཁ͕͋Δ
·ͱΊ • ίϯςφΛಈ͔͚ͩ͢ͷϗετͰ͋ΕCoreOS ͪΐ͏Ͳ͍͍େ͖͞ • ेʹγϯϓϧ • γϯϓϧ͚ͩʹߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ͋Δ • ຊʹKubernetesΛࣗͰϚωδϝϯτ͢Δͷʁ