Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Testing with microservices in merpay
Search
kazegusuri
April 14, 2018
Programming
10
10k
Testing with microservices in merpay
Go Conference 2018 Spring
kazegusuri
April 14, 2018
Tweet
Share
More Decks by kazegusuri
See All by kazegusuri
go-sqlite3を使ってCloud Spannerエミュレーターを作ってみた / Cloud Spanner emulator with go-sqlite3
kazegusuri
5
6.6k
handy-spanner GCPUG
kazegusuri
4
1.9k
Open SKT: メルペイ開発の裏側 / builderscon tokyo 2019 Open SKT
kazegusuri
22
27k
Keep watching and extending features of gRPC
kazegusuri
3
2.5k
Real World Mercari API Architecture
kazegusuri
1
6.2k
gRPC and REST with gRPC in practice
kazegusuri
19
7.9k
Fluentdで始めるPrometheus / Prometheus Tokyo Meetup #1
kazegusuri
1
1.9k
GRPCの実践と現状での利点欠点 / Go Conference 2016 Spring
kazegusuri
44
32k
OutputとBufferedOutputの間の何か
kazegusuri
2
3.3k
Other Decks in Programming
See All in Programming
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
190
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
190
Team operations that are not burdened by SRE
kazatohiei
1
210
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
230
関数型まつり2025登壇資料「関数プログラミングと再帰」
taisontsukada
2
850
すべてのコンテキストを、 ユーザー価値に変える
applism118
2
820
AIエージェントはこう育てる - GitHub Copilot Agentとチームの共進化サイクル
koboriakira
0
380
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
420
生成AIコーディングとの向き合い方、AIと共創するという考え方 / How to deal with generative AI coding and the concept of co-creating with AI
seike460
PRO
1
330
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
890
5つのアンチパターンから学ぶLT設計
narihara
1
110
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
530
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Documentation Writing (for coders)
carmenintech
72
4.9k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
930
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
The Language of Interfaces
destraynor
158
25k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Embracing the Ebb and Flow
colly
86
4.7k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.3k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
VelocityConf: Rendering Performance Case Studies
addyosmani
330
24k
Transcript
Go Conference 2018 Spring Masahiro Sano (@kazegusuri) Testing with microservices
in merpay
Agenda • ࣗݾհ • ϝϧϖΠͷܾࡁγεςϜ • ςετϑϨʔϜϫʔΫ • ςετͷछྨ εϥΠυ
Speaker Deck ʹ͋Γ·͢ʂ https://speakerdeck.com/kazegusuri
ࣗݾհ • ࠤ ਖ਼ߒ (@kazegusuri) • Mercari, Inc → Souzoh,
Inc → merpay, Inc • Principal Software Engineer • Backend Engineer • GoͰܾࡁγεςϜશൠΛ։ൃத • ͖ͳ͜ͱ • gRPC
@kazegusuri Go Conference 2016 Spring GRPCͷ࣮ફͱ ݱঢ়Ͱͷརɾܽ
ϝϧΧϦʹ͓͚ΔgRPC • 2015ࠒʹϝϧΧϦͰॳΊͯgRPCΛಋೖ • ͜ΕܾࡁγεςϜͷͨΊͷϚΠΫϩαʔϏεͷҰͭ • ͦͯ͠ϝϧϖΠ
None
None
ϝϧϖΠͷٕज़ελοΫ • Go + gRPC • Microservices Architecture • Kubernetes
(GKE) + (EKS?) • ͦͷଞ • CircleCI, Spinnaker • Stackdriver Logging, Profiler • Datadog (Monitoring, Tracing) • Pagerduty, Sentry
ܾࡁͷٕज़తͳ͠͞ͱָ͠͞ • ܾࡁ͍͠ (ͱݴΘΕΔ) • ܾࡁϓϩόΠμͱͷҰ؏ੑ • ΫϦςΟΧϧ • 0.1%ͷΤοδέʔεΛߟྀͨ͠γεςϜ
• ΤϯδχΞϦϯάͰղܾ
ϚΠΫϩαʔϏεͱҰ؏ੑ • Microservices Architecture is super difficult!! • Monolithic ArchitectureͰτϥϯβΫγϣϯͰ݁
• Microservices ArchitectureͰඞવతʹτϥϯβΫγϣϯ • ݁Ռ߹ੑ • Idempotency • Retry, retry, retry • Async, Message Queue
ϝϧϖΠͷܾࡁγεςϜ • Microservices Architecture • ػೳຖʹϚΠΫϩαʔϏεΛߏங • த֩ͱͳΔܾࡁγεςϜ͕֤ϚΠΫϩαʔϏεͷঢ়ଶΛཧ • ঢ়ଶભҠϞσϧ
• ঢ়ଶ͕֬ఆ͢Δ·ͰແݶʹϦτϥΠ • ͲΜͳλΠϛϯάͰམͪͯ࠶։Մೳ
ϝϧϖΠͷܾࡁγεςϜ Payment Point Sales Microservices Payment Provider 1 2 3
4 Init 1 2 3 4 Fin
Payment Point Sales Microservices Payment Provider 1 2 3 4
Init 1 2 3 4 Fin Keep retrying with delay ϝϧϖΠͷܾࡁγεςϜ
ਓྨ͕ਖ਼࣮͘͢͠Δͷ͍͠
Ͳ͏ͬͯςετ͢Δ͔ • ڞ௨ͷςετϑϨʔϜϫʔΫΛ࡞ • ֤ঢ়ଶຖʹ͋Δ͖ঢ়ଶ(σʔλͷଘࡏɺ)Λఆٛ • ঢ়ଶભҠͤ͞ͳ͕Βঢ়ଶΛ֬ೝ • ࠷ऴతʹΰʔϧ(डཧঢ়ଶ)ʹͳΔ •
͜ͷςετΛਖ਼ৗܥ͚ͩͰͳ༷͘ʑͳҟৗঢ়ଶΛద༻ͯ͠ਖ਼͘͠ಈ࡞͢ Δ͔֬ೝ͢Δ
ςετϑϨʔϜϫʔΫͷαϯϓϧίʔυ
Failure Injection • ֎෦௨৴ʹରͯ͠ҙͷλΠϛϯάͰΤϥʔΛൃੜͤ͞Δ • MySQL • ֎෦αʔϏεͷHTTPϦΫΤετ • ෦αʔϏεͷgRPCϦΫΤετ
• ༷ʑͳΤϥʔύλʔϯͰঢ়ଶભҠ͕ਖ਼͘͠ߦΘΕΔ͔֬ೝ
Failure Injection (MySQL) • ΤϥʔλΠϛϯά • Query • Execͷྃޙ •
Commitͷྃޙ • શͯͷMySQL queryʹରͯ͠hook͢Δ • github.com/shogo82148/go-sql-proxy • ༷ʑͳhook͕ࠐΊͯศར!
Failure Injection (MySQL)
Failure Injection (HTTP) • ΤϥʔλΠϛϯά • HTTPϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
net/http.RoundTripper ʹΑΔ࣮
Failure Injection (gRPC) • ΤϥʔλΠϛϯά • gRPCϦΫΤετͷྃޙ • ಛఆͷαʔϏεʹର͢Δ௨৴શͯhook͢Δ •
Client InterceptorʹΑΔ࣮ • ·ͩಋೖͰ͖͍ͯͳ͍
Random or Deterministic? • Ͳ͏͍͏݅ͰΤϥʔΛൃੜͤ͞Δ͔ • ΤϥʔΛϥϯμϜͰൃੜͤ͞Δͱςετ͕ෆ҆ఆʹͳΔ • Τϥʔ֬Λ͋͛Δͱޭ͠ͳ͘ͳΓςετ͕࣌ؒ͘ͳΔ •
Τϥʔͷൃੜܾఆతʹߦ͍͍ͨ • stacktrace(runtime.Callers) Ͱܾఆ • Ұൃੜͨ͠ύεͰ2ͱൃੜͤ͞ͳ͍
αʔϏεͷϦΫΤετͷςετ • ςετͰ֎෦ɾ෦αʔϏεΛ͏ͷ͍͠ • ϩʔΧϧͰͷߏங(docker)ɺσʔλͷॳظԽɺςετؒͷґଘ • Mock୯७ͳUnitςετʹྑ͍ • ϦΫΤετʹର͢ΔϨεϙϯε͕ݻఆ͞Εͨঢ়ଶͰͷϦάϨογϣϯ ςετ
• ༧ଌෆೳͳঢ়ଶͰͷςετ
Fake Implementation • αʔϏεͷ୯७Խ͞Ε࣮ͨ • σʔλϕʔεͳͲʹґଘͤͣʹߴʹಈ࡞ • ϥΠϒϥϦଆͰఏڙ͍ͯ͠Δͷ͋Δ • Google
Cloud Pub/Sub • ༻్ʹΑ༷ͬͯʑͳϨϕϧͰ࡞͢Δ • αʔϏεɺpackage, interface
Google Cloud Pub/Sub ͷ Fake
Conformance Test for Fake • FakeͷৼΔ͍͕ຊདྷͷ࣮ͱဃ͍ͯ͠ͳ͍͔ • ظ͢ΔৼΔ͍Λςετέʔεʹ͢Δ • ຊདྷͷ࣮ͱFakeͷ྆ํʹର࣮ͯ͠ߦ
None
ςετͷछྨ • Unit Test / Integration Test • E2E Test
• Chaos Test • QA
Unit and Integration Test • packageͰߦ͏ςετ • ݫີͳ୯ମςετʹ͍ͯ͠ͳ͍ • ԼҐϨΠϠ·ͰؚΊͨςετ
• ґଘinterfaceશͯΠϯδΣΫγϣϯ͢Δ • ґଘαʔϏεͷଓFakeΛར༻ • σʔλϕʔεͷڞ༗͕
σʔλϕʔεͷςετؒͷಠཱੑ • ֤ςετͰDB·Ͱଓ͢ΔͱpackageؒͷςετͰ่յ͢Δ͜ͱ͕ଟʑ • DB͕ڞ༗͞Ε͍ͯΔ͜ͱ͕ • ͳΒDBΛςετຖʹཱͯΕྑ͍ʂ • TestMainͰpackageຖʹಠཱͨ͠databaseΛ࡞(instanceͰͳ͍) •
1ϑΝΠϧʹ·ͱΊͨschemaΛ༻ҙ • multiStatements=trueʹ͢ΔͱExecͰෳΫΤϦΛ·Δ͝ͱ͛ΒΕΔ • ࡞ʹ͔͔Δ࣌ؒ1ඵ΄Ͳ
E2E Test • ࣮ࡍʹαʔόΛىಈͯ͠ϦΫΤετΛ͛ͯৼΔ͍Λςετ • ՄೳͳݶΓґଘαʔϏεىಈ͢Δ • docker-composeͰڥߏங • ςετؒͷಠཱੑςετηοτຖʹಠཱͨ͠σʔλΛ࡞Δ
• ͜͜·ͩਏ͍ͱ͜Ζ͕ଟ͍ • E2EςετͰෳࡶͳςετΛ࣮ࢪ • ಉ࣌ϦΫΤετڝ߹͢ΔΑ͏ͳϦΫΤετ
Chaos Test (Trying) • github.com/Shopify/toxiproxy • framework for simulating network
conditions • Ԇͤͨ͞ΓଳҬ੍ݶͨ͠Γམͱͨ͠ΓͰ͖Δ • gRPCͩͱ͍Ͳ͜Ζ͕͍͠ • ࠓͷͱ͜ΖԆΛൃੜͤͯ͞ݱ࣮తͳϦΫΤετͷϨΠςϯγͷ ༧ଌʹར༻
QA (Quality Assurance) • PostmanΛͬͨAPIςετ • JSONͰϦΫΤετ͢ΔͨΊʹgrpc-gatewayΛར༻ • PostmanͷςετέʔεGitHubͰཧͯ͠CI •
QAͷਓ͕ࣗߦ͍ͬͯΔ!
·ͱΊ • ϚΠΫϩαʔϏε͍͠ • ςετϑϨʔϜϫʔΫͰςετ߲ͷύλʔϯԽ • Failure Injection TestͰΤοδέʔεͷόάݟ͚ͭ͘͢ •
MockΑΓFake
We’re Hiring! https://www.merpay.com/jp/careers/index.html