Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of aut...
Search
moznion
August 18, 2018
Technology
21
7.2k
無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of autonomous and infinite scalable Jenkins with AWS
Kyotoなんか #4の資料です
moznion
August 18, 2018
Tweet
Share
More Decks by moznion
See All by moznion
Chrome Extension Techniques from Hell
moznion
1
160
Simple組み合わせ村から大都会Railsにやってきた俺は / Coming to Rails from the Simple
moznion
4
7k
AIレシート読み取り機能をRuby on Rails on AWSで実現するLLMにまつわるアレコレ / AI-based receipt reading function powered by LLM on Ruby on Rails on AWS
moznion
3
770
Develop to Survive - YAPC::Hakodate 2024 Keynote
moznion
10
17k
これはPerl? それともRuby? クイズ〜〜〜〜〜!!!- Perl or Ruby Quiz
moznion
3
2.7k
Perl 5 OOP機構30年史 - Perl 5's OOP Mechanism over the past 30 years
moznion
1
1k
RuboCopのカスタムCopを書いてContributionしてみる - Contributing a Custom Cop to RuboCop: A Hands-on Experience
moznion
0
83
AWS上に構築する メンテ容易なElasticsearch System / Maintainable Elasticsearch system on AWS
moznion
2
3.9k
PROXY Protocol aware Proxy Server on Node.js
moznion
2
2.5k
Other Decks in Technology
See All in Technology
クォータ監視、AWS Organizations環境でも楽勝です✌️
iwamot
PRO
1
240
IVRyにおけるNLP活用と NLP2025の関連論文紹介
keisukeosone
0
180
LiteXとオレオレCPUで作る自作SoC奮闘記
msyksphinz
0
210
こんなデータマートは嫌だ。どんな? / waiwai-data-meetup-202504
shuntak
6
1.8k
50人の組織でAIエージェントを使う文化を作るためには / How to Create a Culture of Using AI Agents in a 50-Person Organization
yuitosato
6
3.3k
NLP2025 参加報告会 / NLP2025
sansan_randd
4
530
OpenSearchでレガシーな検索処理の大幅改善をやってやろう
dznbk
1
140
自分の軸足を見つけろ
tsuemura
2
620
Ops-JAWS_Organizations小ネタ3選.pdf
chunkof
2
120
ElixirがHW化され、最新CPU/GPU/NWを過去のものとする数万倍、高速+超省電力化されたWeb/動画配信/AIが動く日
piacerex
0
110
LangfuseでAIエージェントの 可観測性を高めよう!/Enhancing AI Agent Observability with Langfuse!
jnymyk
0
180
MCP Documentation Server @AI Coding Meetup #1
yyoshiki41
2
2.6k
Featured
See All Featured
Making the Leap to Tech Lead
cromwellryan
133
9.2k
The Cost Of JavaScript in 2023
addyosmani
49
7.7k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Testing 201, or: Great Expectations
jmmastey
42
7.4k
How GitHub (no longer) Works
holman
314
140k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
30k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
135
33k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2k
Writing Fast Ruby
sferik
628
61k
How to Ace a Technical Interview
jacobian
276
23k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.2k
Transcript
ແݶʹεέʔϧ͢Δ্ʹ ࣗతͳJenkinsʹݟΔເ ~AWSἫ~ KyotoͳΜ͔ #4 @moznion
@moznion Software engineer Hatena Intern 2013
Jenkins
Jenkins - CI (Continuous Integration) Platform - Workflow - ศརͳcron
Why Jenkins?
Why Jenkins? - ͱͱ (Ұ෦ࠓ) CircleCIΛ͍ͬͯͨ - Job͕૿͑ΔʹͭΕ٧·Δqueue - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
- ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ͍͑ͨ -
Why Jenkins? - ͱͱ (Ұ෦ࠓ) CircleCIΛ͍ͬͯͨ - Job͕૿͑ΔʹͭΕ٧·Δqueue - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
- ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ͍͑ͨ - ͱͱJenkinsΛεέʔϧͤ͞Δํ๏Λ͍ͬͯͨ
Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢
Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢ - ฒྻ࣮ߦJobͷ૿Ճʹ͏ੑೳྼԽ - ߜΔͱbuildͪͷΩϡʔ͕٧·Δ͠…… - Slave nodeͷӡ༻େม͗͢Δ - Master
InstanceࢮͶͳ͍ - ͓ئ͍͠·͢ͷڪා (™ @kyanny)
Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢ - ฒྻ࣮ߦJobͷ૿Ճʹ͏ੑೳྼԽ - ߜΔͱbuildͪͷΩϡʔ͕٧·Δ͠…… - Slave nodeͷӡ༻େม͗͢Δ - Master
InstanceࢮͶͳ͍ - ͓ئ͍͠·͢ͷڪා (™ @kyanny)
ฒྻjob্͛ΔͷϚδَʂ
ฒྻjob্͛ΔͷϚδَ SlaveΛ͢ͷ؆୯ʂʂʂʂ
ฒྻjob্͛ΔͷϚδَ - slave͢ͷ؆୯ͳΜ͚ͩͲ…… - ୯७ʹཧίετ͕૿͑Δͧʂ - ʮ͜ͷslaveͩͱͳ͔ͥςετ͕ίέ·͢ʯ
ฒྻjob্͛ΔͷϚδَ - slave͢ͷ؆୯ͳΜ͚ͩͲ…… - ΦʔτεέʔϦϯάΛͪ·ͪ·͍ͬͯ͘ඞཁ͕͋Γ…… - ΠϯελϯεΛ༡͓ͤͯ࣌ؒ͘ແ͍ - CIΑ΄ͲϓϩδΣΫτཱ͕ͯࠐΜͰͳ͍ݶΓ৸ͯΔ͕࣌ؒ ࢧత
- ׂͱઃఆ͕͍͠ (ίπ͕͍Δ)
͏͏……
զʑ΄ͬͱ͍ͯແݶʹϏϧυ͕εέʔϧ͢Δ Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……
ͦΜͳͷ…… ͋Δʂʂʂ
ͦΜͳͷ…… ͋Δʂʂʂ
AWS CodeBuild
AWS CodeBuild - ୯Ґͷ࣌ؒି͠ - ҙͷίϯςφΛ࣮ߦͰ͖Δ - ͦͷίϯςφ্ͰϏϧυΒςετΒ͢Δ - (LambdaΑΓ͕͔͔࣌ؒΔλεΫͷ࣮ߦڥʹ……)
AWS CodeBuild - Jenkins Plugin͕͋Δ (AWSۘ) - https://github.com/awslabs/aws-codebuild-jenkins-plugin - JenkinsͰड͚ͨτϦΨͰCodeBuildΛൃՐͰ͖Δ
https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/ ࣮࣭ແݶʂʂʂʂ
ͨΒ͞ΕͨԸܙ - slave nodeͷϝϯςΒͣ - ΨϯΨϯฒྻϏϧυ͢Δͧʂʂʂ - ࣌ؒି͠ (= ͚ͬͨͩ)
ͷྉۚઃఆͳͷͰ ࣗͰΦʔτεέʔϧॲཧΛ͠ͳͯ͘Α͍
Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢ - ฒྻ࣮ߦJobͷ૿Ճʹ͏ੑೳྼԽ - ߜΔͱbuildͪͷΩϡʔ͕٧·Δ͠…… - Slave nodeͷӡ༻େม͗͢Δ - Master
InstanceࢮͶͳ͍ - ͓ئ͍͠·͢ͷڪා (™ @kyanny)
Master Instance͕ਆ֨Խ͢Δ
Master Instanceਆ֨Խ - Jenkinsͷmaster datamasterϚγϯͷ*ϩʔΧϧετϨʔδ*ʹ อଘ͞ΕΔ - Πϯελϯε͕ࢮΜͩΒऴΘΓ - ͔͠͠ࢮආ͚ΒΕͳ͍
- ͔ͩΒόοΫΞοϓΛऔͬͨΓ͢Δ - e.g. https://papix.hatenablog.com/entry/2016/02/09/101918 - όοΫΞοϓ͔ΒͷϦετΞ࣮ࡍ໘ - ָ͍͠ࣄͰͳ͍…… - μϯλΠϜग़Δ͠……
Master Instanceਆ֨Խ - ݁Ռͱͯ͠ຖଉࡂΛفΔʑ…… - ӡѱ͘Πϯελϯε͕ࢮΜͩΒσΠϦʔόοΫΞοϓ ͔ΒϦετΞ - σΠϦʔόοΫΞοϓͱࢮ͵ؒࡍͱͷࠩʹ͍ͭͯ ఘΊΔ͔͠ແ͍……
͏͏……
զʑৎͰεέʔϧ͢Δ Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……
ͦΜͳͷ…… ͋Δʂʂʂ
ͦΜͳͷ…… ͋Δʂʂʂ
Amazon Elastic File System
AWS Elastic File System (EFS) - AmazonͷnfsΈ͍ͨͳͭ - Instanceʹmountͯ͠͏ -
EBS + nfsతͳ…… - ϘϦϡʔϜαΠζ͕ࣗಈͰ֦ுɾॖখ͢Δ - Disk fullΒͣͷੜ׆
AWS Elastic File System (EFS) - $JENKINS_HOMEΛEFSϘϦϡʔϜ্ʹ͢Δ - Πϯελϯε͕ࢮΜͰ৽͍͠Πϯελϯεʹ ͦͷϘϦϡʔϜΛϚϯτ͢Δ
- => ΠϯελϯεલੈͷهԱΛҾ͖ܧ͗෮׆ʂʂʂ
mount Working...
❌mount Instance down...
❌ New instance coming up...
❌ mount Works fine!!
AWS Elastic File System (EFS) - EBSͰྑ͍ͷͰʁ - ͍ʂɹEBSͷ΄͏͕͍҆͠ʂʂ -
ͨͩEFSϘϦϡʔϜࣗಈ֦ு͕خ͍͠ - (͏͔ͬΓdisk fullʹͳΒͳ͍) - ͋ͱEFSͩͱmulti masterΈ͍ͨͳ͜ͱͰ͖Δ (ޙड़)
Jenkins Multi Master - ଟͰ͖Δ (࣌ؒӡ༻ͯ͠ͳ͍) - ͔͠͠CodeBuildͱڞଘ͍ͯ͠ΔڥͰMulti Masterʹ͢Δ ϝϦοτ΄΅ͳ͘ͳ͍Ͱ͔͢ʁ
- σϓϩΠ࣌ʹҰॠ2masterΛཱ͓͍ͯͯͯɼݹ͍ํΛ ࡴ͢Έ͍ͨͳμϯλΠϜσϓϩΠ͚ͷ͍ํՄೳ
ͨΒ͞ΕͨԸܙ - ΧδϡΞϧʹΠϯελϯε͕ࢮΜͰ҆৺ʂ - ͱ͍͑όοΫΞοϓऔͬͯΔ - ݁Ռͱͯ͠ΠϯελϯεͷೖΕସָ͕͑ʹͳͬͨ - ετϨʔδࣗಈ֦ு͢ΔͷͰDisk fullΒͣ
Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢ - ฒྻ࣮ߦJobͷ૿Ճʹ͏ੑೳྼԽ - ߜΔͱbuildͪͷΩϡʔ͕٧·Δ͠…… - Slave nodeͷӡ༻େม͗͢Δ - Master
InstanceࢮͶͳ͍ - ͓ئ͍͠·͢ͷڪා (™ @kyanny)
ࣾձ: Jenkins͓͡͞ΜͷͳΓखෆ
Jenkinsͷӡ༻Λ୭͔ʹԡ͚͠ΔͷΛΊΔ - ୭Ͱӡ༻Ͱ͖ΔΑ͏ʹ͢Δ - શਓຽJenkins͓͡͞Μ/͓͞ΜԽ - σϓϩΠϝϯςۃྗলྗԽ͢Δ
Jenkinsͷӡ༻Λ୭͔ʹԡ͚͠ΔͷΛΊΔ - Amazon Elastic Beanstalk + Docker - Dockerfile +
Dockerrun.aws.jsonʹΑΔߏͷίʔυԽ - .ebextensionsʹΑΔࣗಈԽ - EFS mountͳͲ - https://github.com/moznion/jenkins_eb_efs_codebuild_example
- Elastic Beanstalkͷmanaged updateͰࣗಈతʹύον͕ͨΔ - ͍ͭͰʹΠϯελϯεͷೖΕସ͑ߦΘΕΔ - EFSڥԼͳͷͰΠϯελϯεೖΕସָ͑ʑࣗಈԽ - جຊతʹӡ༻์ஔͯ͠·͢
- EFSͷϘϦϡʔϜࣗಈ֦ுͷύϫʔͰ͋Δ ͨΒ͞ΕͨԸܙ
·ͱΊ
·ͱΊ - JenkinsͷslaveϊʔυΛAWS CodeBuildʹҕৡ͢Δ͜ͱͰ࣮࣭ ແݶʹฒྻϏϧυ͕εέʔϧ͢ΔΑ͏ʹͳΓ·ͨ͠ - JenkinsͷετϨʔδΛEFSʹ͢Δ͜ͱͰϚελʔσʔλͷ ϩόετԽΛߦ͍·ͨ͠ - Elastic
BeanstalkΛ༻͍ͯJenkinsΛӡ༻͢Δ͜ͱͰӡ༻ͷ ίετԽΛ͠·ͨ͠ - Α͔ͬͨͰ͢Ͷ
Q?
Wait...
͜ͷJenkinsʹ͕͋Δ - EFS͍ʂʂʂʂʂ - Jenkins͕git clone͢ΔͱϚδ͔͔࣌ؒΓ·͢ - git cloneΛճආ͢Δํ๏͍·ͷͱ͜Ζແ͍…… -
shallow cloneΛ༗ޮʹ͏ - sparse checkoutΛ༗ޮʹ͏ - ͳͲͷ͕ඞཁ - ͦͦCodeBuild͕git clone͢ΔΜͩ͠jenkins͠ͳͯ͘ ྑ͍Μ͚ͩͲ…… - ͋Δ͍EBSΛ͏……
ຊʹJenkinsඞཁʁ - ͭ͡AWS CodeBuild࠷ۙΊͬͪΌͰ͖͕Αͯ͘ɼ ୯ମͰे͑Δ - GitHubͷhookड͚ΕΔ͠ - Commit statusΛม͑Δػೳ·Ͱ͍͍ͭͯΔ
- ϏϧυཤྺҰཡͰ͖Δ͠ - CloudWatch LogsʹϏϧυͷϩάग़Δ - JenkinsෆཁͰʁʁʁʁɹ͜ΕͰेͰʁʁʁʁ - ͍
ຊʹJenkinsඞཁʁ - ࣗ༝ͳWebhookΛड͚Δඞཁ͕͋Δͱ͔…… - ͋ͱ௨ͱ͔Ͷʂ - ͋Δ͍ͬͱଞʹϏϧυύΠϓϥΠϯΛܨ ͛Δඞཁ͕͋Δͱ͔…… - ͦ͏͍͏ͱ͖ΜΓͰ͢
- (͋ͱ·͋Ұཡੑ͕Α͍) (ॾઆ͋Γ·͢)
Q?