Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Microservices on Multi-Cloud
Search
kazeburo
March 23, 2018
Technology
21
5.8k
Microservices on Multi-Cloud
MANABIYA TERATAIL DEVELOPER DAYS 2018-03-23
kazeburo
March 23, 2018
Tweet
Share
More Decks by kazeburo
See All by kazeburo
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
13
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
1.8k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
12k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.3k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
560
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.2k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
13k
sacloudns
kazeburo
2
320
Other Decks in Technology
See All in Technology
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
5
760
TypeScript、上達の瞬間
sadnessojisan
46
13k
Platform Engineering for Software Developers and Architects
syntasso
1
520
Flutterによる 効率的なAndroid・iOS・Webアプリケーション開発の事例
recruitengineers
PRO
0
120
安心してください、日本語使えますよ―Ubuntu日本語Remix提供休止に寄せて― 2024-11-17
nobutomurata
1
1k
【Startup CTO of the Year 2024 / Audience Award】アセンド取締役CTO 丹羽健
niwatakeru
0
1.3k
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
複雑なState管理からの脱却
sansantech
PRO
1
150
心が動くエンジニアリング ── 私が夢中になる理由
16bitidol
0
100
テストコード品質を高めるためにMutation Testingライブラリ・Strykerを実戦導入してみた話
ysknsid25
7
2.7k
Shopifyアプリ開発における Shopifyの機能活用
sonatard
4
250
Why App Signing Matters for Your Android Apps - Android Bangkok Conference 2024
akexorcist
0
130
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
459
33k
Designing the Hi-DPI Web
ddemaree
280
34k
GitHub's CSS Performance
jonrohan
1030
460k
Designing for humans not robots
tammielis
250
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
6
420
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
YesSQL, Process and Tooling at Scale
rocio
169
14k
Adopting Sorbet at Scale
ufuk
73
9.1k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
What's in a price? How to price your products and services
michaelherold
243
12k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
Microservices on Multi-Cloud Masahiro @kazeburo Nagano MANABIYA Teratail developer days
2018/03/23
Me • խ • @kazeburo • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability
Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ • झຯDBͷ Restore
Agenda • ϝϧΧϦʹ͍ͭͯ • ϝϧΧϦͷ Infrastructure History #1 - Multi-Cloud
• ϝϧΧϦͷ Infrastructure History #2 - Microservices on Multi-Cloud • Microservices on Multi-Cloud ͷ՝
None
ϝϧΧϦ • ຊ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ
ถࠃ/ӳࠃ ͷల։ JP UK US
KPI μϯϩʔυ GMV(૯औҾֹ) 1ԯDLҎ্(JP+US+UK) ݄ؒ100ԯԁҎ্ ग़ 1100ສҎ্
γεςϜ֓ཁ ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services
LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ը૾ ܾࡁ AI ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏
Infrastructure
Infrastructure in 2017 DNS: Amazon Route53 CDN: Akamai, CloudFront Storage:
Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel JP UK US
Infrastructure in 2018 DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux(JP)
Storage: Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog JP UK US + +
Infrastructure History #1 2013 - 2017 / Multi-Cloud
Infrastructure History (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷʮ͘͞ΒͷVPSʯ1ʹWebDBࡌͤͨߏͰ։࢝ •
Infrastructure ઐऀ͕͍ͳ͍தͰɺ։ൃऀʹۙͳج൫Λબ • ϦϦʔεޙ2ϲ݄Ͱʮ͘͞ΒͷΫϥυʯʮઐ༻αʔόʯҠߦ
ʮ͘͞Βͷઐ༻αʔόʯ • Metal as a Service • ཧαʔόΛΫϥυͷΑ͏ʹѻ͑Δ • ཧαʔόͳΒͰͷύϑΥʔϚϯε
• ωοτϫʔΫͱϋʔυΣΞͷอक ͘͞ΒΠϯλʔωοτ༷͕୲ • ʮ͘͞ΒͷΫϥυʯͱଓ͕Մೳ • ίετύϑΥʔϚϯεʹ༏ΕΔ
Infrastructure History (2) • 2014/09 US ϦϦʔε • AWS (Oregon)
ʹͯαʔϏεߏங • JPϦϦʔε͔Β͠Β͘ܦͪɺ։ൃऀʹAWSܦݧऀ͕૿Ճ • ͦΕͰ Infrastructure ઐऀগͳ͘ɺRDSElastiCacheϚωʔδυαʔϏεΛ ར༻ͯ͠αʔϏεΛߏங • USࠃͷ MaaS Λݕ౼͕ͨ͠ɺUSͰͷαʔϏεͷ༧͕͘͠ɺΫϥυͷॊ ೈ͞Λ JP ΑΓॏཁࢹ
Infrastructure History (3) • 2015/11 SREνʔϜൃ • JP/US ͷΞʔΩςΫνϟΛվળ͠ɺαʔϏεͷ৴པੑͱεέʔϥϏϦςΟͷ ্ʹͱऔΓΉ
• 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़ͱͯ͠ʮGCPʯ্ͰαʔϏεΛߏங
Multi-Cloud in 2017/03 JP UK US ઐ༻αʔό EC2 GCE IaaS
Λத৺ͱͨ͠ Multi-Cloud (Hybrid Cloud) ͨͩ͠ɺͦΕͧΕͷαʔϏεΛΈΔͱ୯ಠͷCloudΛར༻
Multi-Cloud Operations • ՄೳͳݶΓڞ௨ͷΞʔΩςΫνϟΛ࠾༻ • ଞͷΫϥυʹଘࡏ͠ͳ͍ϚωʔδυαʔϏεͷϦϓϨΠε • Consul/Local DNSͷಋೖ •
ΦϖϨʔγϣϯͷڞ௨ԽɾগਓͰͷӡ༻ͷ࣮ݱ • JP ͷنͰ࣮ͷ͋ΔߏɻUS AppStoreͰ3Ґ࣌ͷτϥϑΟοΫΛ҆ఆͯ͠ॲཧ • Ansible playbookɺDBͷϚΠάϨʔγϣϯ࡞ۀͷڞ௨Խ
Architecture nginx nginx nginx DNS-RR App App App App App
App MySQL MySQL memcached memcached util util cloud cloud JP nginx nginx nginx App App App App App App MySQL MySQL memcached memcached util util GCE cloud load balancer GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE GCE UK γϯϓϧͳ3ߏ ΫϥυͰEC2/GCE (αʔό) Λ த৺ʹߏ ɾ USಠࣗͷαʔϏε খنDBʹ RDSΛ͏͜ͱ UKͰCloud Load BalancerΛར༻
Internal DNS App App App App App App DNS DNS
unbound unbound unbound unbound unbound unbound DNS unbound Consul DNS *.consul *.local • શͯͷαʔόʹunboundΛಋೖ • ϩʔΧϧΩϟογϡʹΑΔύϑΥʔϚϯε্ • resolv.conf ΑΓোʹڧ͍ • αʔϏεͷՄ༻ੑͱॊೈੑΛ֬อ • ΞϓϦέʔγϣϯIPΞυϨεͰͳ͘ϗετ໊Λར༻ • ΞϓϦέʔγϣϯίʔυͷมߋͳ͠ʹߏมߋ͕Մೳ • Internal LBସͱͯ͠consul Λ͔ͭͬͨԽͱෛՙࢄΛଟ༻
Infrastructure History #2 2018 - / Microservices on Multi-Cloud
Microservices • αʔϏεͷ Resilience Λ্ͤ͞Δ • ࡉ͔͍୯ҐͰͷεέʔϦϯάɺোͷ • νʔϜɾ৫ͷ Scalability
ΛߴΊΔ • 1000໊Ҏ্ͷΤϯδχΞ৫Λࢤ • αʔϏε։ൃͷΛ͞Βʹ͍͋͛ͯͨ͘Ί
US Re-Architecture • US marketʹΑΓ࠷దԽ͘͢ Client ΛFull Renewal • MicroservicesͷroutingΛߦ͏API
GatewayΛGolangͰ࣮ • AWS্ͷMonolith APIΛWrap • ؇͔ͳҠߦΛ࣮ݱ API Gateway search personalization offer gRPC JSON over HTTPs Protocol Buffers over HTTPs gRPC gRPC Monolith API
API Fork • 3ͭͷRegionͰڞ༗͍ͯͨ͠Monolith APIͷίʔυΛ US,UK ͱ JP Ͱ •
ࣗregionͷมߋ͕ଞregionʹӨڹ͢Δ͜ͱΛ͑ΔɻௐɾQAίετݮ • ΑΓ֤ࠃͷࣄʹ͋ͬͨ։ൃΛ֤ࠃͰߦ͏ • US,UKͷݱ࠾༻ਐల
API Gateway in JP • Monolith API͔ΒݺΕΔ Microservices ͢Ͱʹӡ༻த •
JPͰMicroservicesΛ͞ΒʹਐΊΔͨΊ API GatewayΛಋೖ • Golang͕ͩɺUSͱҟͳΔ࣮ • Clientͷมߋͳ͘Protocolҡ࣋ • DNS cacheɺRequest bufferingͳͲͷՃ API Gateway JSON over HTTPs JSON over HTTPs ServiceA ServiceC ServiceB
Infrastructure in 2018 JP UK US + + ͦΕͧΕͷRegionʹ͋Θͤͨ Microservices
on Multi-Cloud
Microservices Tech Stack • Container / Docker • Kubernetes •
Spinnaker
Container / Docker • Container • Ϧιʔεͷɾ੍ޚ • VMΑΓܰྔͳOSڥΛ࣮ݱ •
Docker • ϙʔλϏςΟͷ࣮ݱ • DockerfileʹΑΔҰ؏ͨ͠Πϝʔδͷ࡞
Container use case Github PR Daily job BigQuery (app-log) index
Container Registory DEPLOY!! Application͚ͩͰͳ͘ MLRecommendͷσʔλΛؚΉContainerΛ࡞ ෳࡶͳMiddleware҆ఆͯ͠ఏڙ container for keyword suggest service
Kubernetes • Container ͷ Orchestration Platform • ࣗಈScalingɺࣗಈhealing • Container
ӡ༻ίετͷݮ • GKE(Google Kubernetes Engine) Λத৺ʹར༻ • k8s͕MicroservicesͷKey factor • AWS EKS/Fargateͷݕূ • ͘͞ΒͷΫϥυɺk8s on Metalͷݕ౼ɾݕূ
Spinnaker • Continuous Delivery Platform • Developed by Netflix •
googleͳͲͷڠྗɾOSSԽ • Deploy pipelineΛఆٛ͠ɺࣗಈ࣮ߦ͢Δ • Multi-Cloud ରԠ • k8s, ECS, OpenStack... • SpinnakerʹΑΔContinuous Delivery http://tech.mercari.com/entry/2017/08/21/092743
Microservices on Multi-Cloud ͷ՝
Microservices on Multi-Cloud Pros/Cons • Pros: Service ʹద࣮ͨ͠ߦڥͷબ • σʔλϕʔεɾMLܥαʔϏεͳͲ৽͍ٕ͠ज़Λૉૣ͘औΓࠐΉ
• ։ൃऀ͕ٕज़બݖΛͭ͜ͱͰɺΦʔφʔγοϓΛΑΓڧ͘ • Cons: Ϋϥυؒ࿈ܞͷޮੑ • ωοτϫʔΫίετ • Ϋϥυؒͷڑ • Cons: αʔϏεͷՄ༻ੑҡ࣋
Distance between clouds ੴङ DC Cloud Service Mircoservices Infrastructure ઐ༻αʔό
Monolith API Infrastructure 1,000 km
Distance between clouds $ ping -c 3 example.mercari.jp PING example.mercari.jp
(x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=50 time=18.6 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=50 time=18.4 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=50 time=20.6 ms ੴङ(ઐ༻αʔό) ▶︎ ౦ژ(Google Cloud Load Balancer) $ ping -c 3 example.mercari.jp PING example.mercari.jp (x.x.x.x) 56(84) bytes of data. 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=1 ttl=56 time=1.09 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=2 ttl=56 time=1.08 ms 64 bytes from x.bc.googleusercontent.com (x.x.x.x): icmp_seq=3 ttl=56 time=1.14 ms ౦ژ(͘͞ΒͷΫϥυ) ▶︎ ౦ژ(Google Cloud Load Balancer) 18-20 ms 1 ms ಉ͡DCͰ͋Ε 0.1 ms
Distance between clouds by HTTPS $ ./httpstat.sh https://example.mercari.jp/hc HTTP/1.1 200
OK Server: nginx/1.13.3 Date: Wed, 11 Oct 2017 01:59:15 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Wed, 11 Oct 2017 02:59:15 GMT Cache-Control: max-age=3600 Cache-Control: public Via: 1.1 google Alt-Svc: clear DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 1ms | 19ms | 165ms | 20ms | 0ms ] | | | | | namelookup:1ms | | | | connect:20ms | | | pretransfer:185ms | | starttransfer:205ms | total:205ms
How to beyond the distance • 3 way handshakeΛආ͚ΔɻTLS ͷ
handshake ආ͚Δ • HTTP/1, HTTP/2 ͷKeepAlive Λ׆༻͢Δ • ChoconͰͷConnection Aggregation
chocon • GoͰ࣮ͨ͠γϯϓϧͳ Proxy Server • OSSͱͯ͠ެ։ • github.com/kazeburo/chocon •
1Ҏ্ͷՔಇ࣮
chocon % curl -H ‘Host: example.com.ccnproxy-https’ http://10.0.0.1/v1/foo *.ccnproxy-https IN CNAME
chocon.local. ෦DNSΛ׆༻͢ΔͱURLͷϗετ໊Λมߋ͢Δ͚ͩ chocon Web Client https://example.com/ ʹproxy http http or https keepAlive Private Network % curl http://example.com.ccnproxy-https/v1/foo
After Chocon $ ./httpstat.sh /dev/null https://microservice.example.com.ccnproxy-https/hc HTTP/1.1 200 OK Cache-Control:
max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms pingͱಉͷ
Durability, Availability • Multi-CloudͰՄ༻ੑԼ͕Δ • ͲͷΫϥυ͕མͪͯαʔϏεͷܧଓʹӨڹ • Քಇ 99.99% ͱ
99.95% ͷΫϥυΛ͍ͬͯΔ߹ɺՔಇ 99.95%ʹͳΔ • MicroservicesͰಛఆͷαʔϏε͕མͪͯશମʹӨڹ͠ͳ͍ͤ͞ͳ͍ • Өڹ͕͑ΒΕΔMicroservicesಛఆͷCloudͰӡ༻ • ߴ͍Մ༻ੑ͕ඞཁͱ͞ΕΔMicroservicesMulti-CloudͰల։
Massive Computing Resource Service Mesh Service Mesh J Infrastructure in
the near future? Security / DDoS mitigation API Gateway A B C D D E CloudA CloudB F CloudC (Monolith API) H K L M
ॊೈͰ৴པੑͷߴ͍ Infrastructure Λ Microservices ͱ Multi-Cloud Ͱ࣮ݱ
We’re Hiring! careers.mercari.com