Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Terraform, GitHub Actions, Cloud Buildでデータ基盤をPr...
Search
Shinichi Nakagawa
PRO
March 15, 2024
Technology
2
3.1k
Terraform, GitHub Actions, Cloud Buildでデータ基盤をProvisioningする / Data Platform provisioning for Google Cloud and Terraform
会社名は入ってますが100%個人の開発と趣味の話です.
Shinichi Nakagawa
PRO
March 15, 2024
Tweet
Share
More Decks by Shinichi Nakagawa
See All by Shinichi Nakagawa
AI・LLM事業部のSREとタスクの自動運転
shinyorke
PRO
0
350
実践Dash - 手を抜きながら本気で作るデータApplicationの基本と応用 / Dash for Python and Baseball
shinyorke
PRO
2
2.5k
Cloud RunとCloud PubSubでサーバレスなデータ基盤2024 with Terraform / Cloud Run and PubSub with Terraform
shinyorke
PRO
9
3.8k
自らを強いエンジニアにするための3つの習慣 / I need to be myself, I can't be no one else
shinyorke
PRO
82
83k
阪神タイガース優勝のひみつ - Pythonでシュッと調べた件 / SABRmetrics for Python
shinyorke
PRO
1
1.4k
Pythonとクラウドと野球の推し活. / Baseball Data Platform for Python and Google Cloud
shinyorke
PRO
2
2.9k
月額コーヒー3.34杯分のコストでオオタニサンの活躍を見守るデータ基盤のはなし / Pyhack Con
shinyorke
PRO
2
500
俺のDXを実現するためのサーバレスなデータ基盤開発と運用 / Serverless Data Platform and Baseball
shinyorke
PRO
5
12k
機械学習エンジニアが目指すキャリアパスとその実話 / My Journey to Become a ML Engineer
shinyorke
PRO
10
18k
Other Decks in Technology
See All in Technology
ここはMCPの夜明けまえ
nwiizo
1
420
DETR手法の変遷と最新動向(CVPR2025)
tenten0727
2
1.3k
AWS Control Towerを 数年運用してきての気づきとこれから/aws-controltower-ops-tips
tadayukinakamura
0
130
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming
tomzoh
1
230
Amazon CloudWatch を使って NW 監視を行うには
o11yfes2023
0
150
AI Agentを「期待通り」に動かすために:設計アプローチの模索と現在地
kworkdev
PRO
2
440
SREの視点で考えるSIEM活用術 〜AWS環境でのセキュリティ強化〜
coconala_engineer
1
290
Would you THINK such a demonstration interesting ?
shumpei3
1
220
OSSコントリビュートをphp-srcメンテナの立場から語る / OSS Contribute
sakitakamachi
0
1.4k
Running JavaScript within Ruby
hmsk
3
320
OpenSearchでレガシーな検索処理の大幅改善をやってやろう
dznbk
2
170
LiteXとオレオレCPUで作る自作SoC奮闘記
msyksphinz
0
590
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.2k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
135
33k
Navigating Team Friction
lara
184
15k
Adopting Sorbet at Scale
ufuk
76
9.3k
What's in a price? How to price your products and services
michaelherold
245
12k
We Have a Design System, Now What?
morganepeng
52
7.5k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
12k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.3k
Being A Developer After 40
akosma
91
590k
Building Flexible Design Systems
yeseniaperezcruz
329
38k
Why Our Code Smells
bkeepers
PRO
336
57k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Transcript
Terraform, GitHub Actions, Cloud BuildͰ σʔλج൫ͷProvisioningΛͬͯΈͨ. αʔόϨεΞʔΩςΫνϟͰMicro ServiceԽͨ͠ੈքઢͷDevOpsతͳߟ. Shinichi Nakagawa
2024/03/15 Jagu'e'r Cloud Native #13 ϋΠϒϦου Meetup
໔ࣄ߲ • ͜ͷࣄྫࢲʢத৳Ұʣݸਓͷझຯ։ൃ͔Βੜ·ΕͨߟͰ͢. • ॴଐ৫ɾاۀɾஂମΛද͢ΔൃݴɾҙݟͰ͍͟͝·ͤΜ. • ॴଐ৫ɾاۀ͓ΑͼͦͷεςʔΫϗϧμʔʹର͢Δ࣭ɾҙݟɾ ͍߹Θ͓ͤΑͼۀ༰ʹҰ͓͑͢Δ͜ͱͰ͖·ͤΜ. • εϥΠυͷը૾ݪଇʮ͍Β͢ͱʯΛར༻͍ͯ͠·͢.
※ʮ͍Β͢ͱʯҎ֎ͷը૾ग़యɾҾ༻ݩΛهࡌ͍ͯ͠·͢.
Who am I ? ʢ͓લ୭Α?ʣ • Shinichi Nakagawaʢத৳Ұʣ / @shinyorkeʢSNSʣ
• ΞΫηϯνϡΞגࣜձࣾϚωδϟʔ • ΫϥυΤϯδχΞϦϯά/SREʢຊ৬ʣ • ʮGoogle Cloud Partner Top Engineer 2024ʯड • AI, ML, Dataplatform, Baseballʢաڈͷࣄ+झຯಛٕʣ • ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ ʮٕज़ʯʮٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢. https://shinyorke.hatenablog.com/
ຊͷ͓͠ͳ͕͖ʢ͢͜ͱʣ • σʔλج൫ΛTerraformͰؙͬͱIaCʢInfrastructure as Codeʣ. • GitHub ActionsͰGoogle CloudΛDevOpsͯ͠Έͨ. Cloud
RunͰ࡞ͬͨMicro ServiceΛͯ͠IaC&DevOpsͨ͠Ͱ͢. ※͜ͷࣄྫۀͰͳ͘झຯͰ͢ʢ͕࣮Ͱ͑Δͣʣ.
ٿσʔλੳج൫ ※ݸਓ։ൃͷͰ͢ • झຯͷٿΛຊ֨తʹσʔλͰݟΔͨΊͷج൫ΛຊؾͰ ࡞͍ͬͯ·͢ʢٕज़తͳݕূΛ݉Ͷͯʣ. • ಛʹϝδϟʔϦʔάʮBaseball Savantʯͱ͍͏ެࣜͷ σʔλαΠτ͕͋Γ, ϑΝϯͷզʑͰCSVܗࣜͰσʔλ
͕खʹೖΓ·͢&σʔλͷத͕݁ߏ͍͜͠Ͱ͢. • ͦͷʮ͍͜͠CSVσʔλʯΛऔΓࠐΈ, ੳɾղੳΛ ߦ͏ϝδϟʔϦʔάੳΞϓϦΛ2022͔Β։ൃɾӡ༻. ຖϦΞʔΩςΫνϟʢ࡞Γ͠ʣ͍ͯ͠·͢, ࠓ Cloud Runͷ৽ػೳʢCloud Run JobsʣͰ࡞Γ͠. ੳج൫ͷΞϓϦέʔγϣϯʢ2022൛ʣ ©Shinichi Nakagawa ©Shinichi Nakagawa
ٿσʔλੳج൫શମ૾ʢ2024൛ʣ ©Shinichi Nakagawa
σʔλج൫ͷ֤ػೳͰͬͯΔϞϊͱղઆ .JDSP4FSWJDFPSׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ DSPOࣜͰͷ5JNFS
5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮ -VJHJ ଞͷ࣮Ͱ0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμϯϩʔυ (Pͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃͷੜσʔλͱͯ͠อ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (Pͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ੳऀPSΞϓϦ͔Β͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻
.JDSP4FSWJDFPSׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ DSPOࣜͰͷ5JNFS 5SJHHFS
8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮ -VJHJ ଞͷ࣮Ͱ0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμϯϩʔυ (Pͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃͷੜσʔλͱͯ͠อ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (Pͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ੳऀPSΞϓϦ͔Β͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻ σʔλج൫ͷ֤ػೳͰͬͯΔϞϊͱղઆ શ෦खಈͰઃఆ͢Δͱഁ͢Δະདྷ͕͍ͬͯ·͢ʂ
Micro ServiceʹTerraformඞཁʢॾઆ༗Γʣ. • ෳͷΞϓϦέʔγϣϯΛचͭͳ͗ʢϐλΰϥεΠονʣ͢Δ߹, TerraformͰͷIaC͓ͬͯ͜͏, খ͍͞γεςϜͰ. • ࠓճͷߏͰ·͋·͋ͷͷGoogle CloudͷServiceΛར༻. •
Cloud RunΞϓϦ͕3ݸ + Volume, LoggingͳͲͷݻ༗ઃఆ • Pub/Sub TopicͱSchema͕ͦΕͧΕ2ݸ • Cloud Schedulerͷઃఆ • Cloud StorageͷBucketͱΞΫηε੍ޚ • ͜ΕΒͱ৭ʑඥͮ͘ઃఆ, ಛʹService Account͓ΑͼIAM←͜Ε͕࠷ۂऀʢηΩϡϦςΟతʹʣ • ҎલCloud Console or GUIͰ࡞͍͕ͬͯͨഁ͢Δʢྫ: ʹ͕֮͑ແ͍Service Account, InstanceͳͲʣͷͰࠓ͔ΒIaCԽ. ઃܭͱ࣮ͷॳظίετʢओʹ࣌ؒͱΩϟονΞοϓʣ͋Δ͕, ࢿͨ͠΄͏͕ઈରʹྑ͍ʢͱ࣮ײ͍ͯ͠·͢ʣ.
Google CloudͰTerraformΛ࢝ΊΔ࣌ • Quick StartͳͲ, ެࣜυΩϡϝϯτʹsnippet͕͋ΔͷͰ͓͏ʂ • gcloud commandGUIઃఆͱͷฒͼʹTerraformͷsnippet͕ଘࡏ. •
ࢲެࣜυΩϡϝϯτͷTerraformͷsnippetΛmain.tfʹషΓ͚, ಈ࡞ ֬ೝ͠ͳ͕Βຊ൪༻ͷ࣮Λߦͬͨʢ͘͢͝ḿͬͨʣ. શͯͷυΩϡϝϯτʹ͋Δ༁Ͱແͦ͞͏͕ͩੵۃతʹ͓͏.
ʁʁʁʮ͍͍ײ͡ʹࣗಈԽͱDevOps͍ͨ͠ΜͰ͕͢Ͳ͏͢Ε🤔ʯ Enterprise͚Ͱ࠷ۙ͋Δͱࢥ͏͜ͷཁʹͲ͏Ԡ͑Δ͔!?
ʲʳGitHub Actions͔ΒTerraformΛͬͨ • ʮ໎ͬͨΒGitHub ActionsʯͱࢥͬͯͬͨΒҙ֎ͱ্ख͍͖͘·ͨ͠. • ͬ͘͟Γॻ͘ͱҎԼͷखॱͰ࣮ݱ. 1. Workload Identityͷ४උ.
2. CDʢܧଓతσϓϩΠʣͱ࣮ͯ͠ߦ͢Δରͷܾఆ. 3. λεΫͷॱ൪ΛܾΊΔ. • Cloud RunΞϓϦέʔγϣϯͷ߹, ͜ͷ͚۠Ͱ͏·͘ߦͬͨ. • Workload Identity͓ΑͼΞϓϦͷService AccountʢSAʣൃߦGitHub ActionsͰͳ͘TerraformͰखಈ࣮ߦ. • SAͱ֤छϦιʔεͷBinding, ͦͷଞͷϦιʔεͷCDGitHub ActionsͰ࣮ߦ. • ArtifactʢDocker imageʣੜ -> DeployΛઈରతͳॱংͱͨ͠GitHub Actions Work fl owͷઃܭͱ࣮. ※͕͜͜ॏཁ
ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Requestཱ͕͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Requestཱ͕͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Requestཱ͕͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Requestཱ͕͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Requestཱ͕͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
GitHub Actions + Terraform, Cloud Buildͷ • CI/CDεςʔδͷׂΛ໌֬Խ. • ʢͨΓલͰ͕͢ʣςετຖճࣗಈ࣮ߦ.
• Pull Request࣌ʹImage Buildͱterraform planͷ֬ೝ. • Deployʢterraform applyʣmainͷMergeͷΈ. • GitHub ActionsͰCI/CDͷϑϩʔΛ੍ޚ. • ͯ͢ͷΞϓϦέʔγϣϯͷArtifactΛCloud BuildͰ࡞ޙʹterraform planΛݩʹϨϏϡʔ. • GitHub ActionsͷneedsઃఆͱPull RequestͷonΠϕϯτͰ͍͍ײ͡ʹ੍ޚ. • ϨϏϡʔ༻ʹterraform planͷ݁ՌΛPull RequestͷίϝϯτʹͤΔ.
݁ͼ • ෳࡶʹͳΓ͕ͪͳMicro ServiceIaCԽ͠·͠ΐ͏, ΕΔͱେม. • GitHub ActionsͱCloud BuildͰTerraformͻͱ͕ඞཁ. ॳखͰIaCʢTerraformʣΛೖΕΔۤ࿑͋Γ·͕͢େมʹॏཁ.
͍͖ͳΓࣄͰΔͱେมͳͷͰԿ͔͠Β࿅शΛ͠·͠ΐ͏.
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠. Shinichi Nakagawa ©Shinichi Nakagawa