Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How we build microservices with Kubernetes & Kafka
Search
Yun Chen
September 26, 2019
Programming
3
380
How we build microservices with Kubernetes & Kafka
An introduce to K8S and Kafka
Yun Chen
September 26, 2019
Tweet
Share
More Decks by Yun Chen
See All by Yun Chen
Redis Lua Script With Red Envelope & Message Queue
chenyunchen
0
600
Kubernetes Library with client-go
chenyunchen
0
410
[Graduate Project] Drupal Upgrade & Docker Deploy
chenyunchen
0
140
WECO Porject
chenyunchen
0
84
From Python Import Fun
chenyunchen
0
400
Docker
chenyunchen
2
190
NISRA fun with burp suite
chenyunchen
1
200
Vim
chenyunchen
0
160
Fun with programming
chenyunchen
0
150
Other Decks in Programming
See All in Programming
カラム追加で増えるActiveRecordのメモリサイズ イメージできますか?
asayamakk
4
1.6k
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
140
約9000個の自動テストの 時間を50分->10分に短縮 Flakyテストを1%以下に抑えた話
hatsu38
23
11k
生成 AI を活用した toitta 切片分類機能の裏側 / Inside toitta's AI-Based Factoid Clustering
pokutuna
0
580
Java ジェネリクス入門 2024
nagise
0
610
Vue SFCのtemplateでTypeScriptの型を活用しよう
tsukkee
3
1.5k
[PyCon Korea 2024 Keynote] 커뮤니티와 파이썬, 그리고 우리
beomi
0
110
go.mod、DockerfileやCI設定に分散しがちなGoのバージョンをまとめて管理する / Go Connect #3
arthur1
10
2.4k
Kaigi on Rails 2024 - Rails APIモードのためのシンプルで効果的なCSRF対策 / kaigionrails-2024-csrf
corocn
5
3.4k
リリース8年目のサービスの1800個のERBファイルをViewComponentに移行した方法とその結果
katty0324
5
3.6k
Generative AI Use Cases JP (略称:GenU)奮闘記
hideg
0
160
Googleのテストサイズを活用したテスト環境の構築
toms74209200
0
270
Featured
See All Featured
Docker and Python
trallard
40
3.1k
Statistics for Hackers
jakevdp
796
220k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Being A Developer After 40
akosma
86
590k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
Adopting Sorbet at Scale
ufuk
73
9k
Ruby is Unlike a Banana
tanoku
96
11k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
328
21k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
290
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
231
17k
Transcript
HOW WE BUILD MICROSERVICES WITH KUBERNETES & KAFKA SDN X
CLOUD NATIVE MEETUP #20 https://speakerdeck.com/chenyunchen/redis-lua-script-with-red-envelope-and-message-queue
WHO AM I Chen Yun Chen (Alex) >
[email protected]
>
blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
KUBERNETES ቘ薹硌獅膏褖ګ
CONTAINER ቘ薹膏褖ګ > Container != Docker > Appᥝெ讕瞥獤ग़㮆Dockerfile? > Log෭扮ইຎ茴褸?
> 緳矒᯿珸ᥝெ讕ڣ䥁? > 瑊絑ह牏܋羷ፘ狅䷱㺔氂? 4
KUBERNETES 硌獅膏褖ګ > Storage: CSI (Container Storage Interface) 吖瑿猋ൔ, 梊秚ګ,
盠ᆙ, ᯿蕦虻碘㳫ᴻ? > Network: CNI (Container Network Interface) ๅṛጱ窕ᰁ, ๅ犵ጱ皤螛, ๅग़ጱ翕᪠Օᶎ? > Computing: CRI (Container Runtime Interface) 犋አDocker? ೭㬵᪒VM? 窾ݳContainer&VM? 5
ܻত礍䯤 > 㻌ݣVM䁆ᤈ㮆誢 > ֵአᒫӣොጱRedis, MySQL๐率 6
疩獈K8S㪔瞥獤๐率 犥㻌Ӟ℄讨傶ֺ 繸ෝ缰ଏ牧蟂℄奞℄ኼ Auth Server > ض疥ጭ獈藨挨๐率ು櫝 Logic Server >
ݳဩጱ藶穩盅蝑狶蒂ቘ 7
疩獈WEBSOCKET蝫娄 Auth Server > 奈TCP蝫娄 ׀Android, iOS藨挨 WebSocket Server >
WebSocket蝫娄 ׀Web藨挨 翕殷粚氃क़螵蜉襑穩 8
瞥獤LOGIC๐率 Http Server > ׀獉蟂ٌਙ㿁褧 篷ဩ蝚螂gRPCොୗፗ蝫Logic Avatar Server > 蕣狶㮆Ո牏ᗭ奲瑽猟
Message Center Server > 懱௳๐率ፘ橕 File Server > 䲆礯๐率ፘ橕牏ݱ圵疳ੑ婘瑽蕣狶 9
KAFKA BUILDING REAL-TIME DATA PIPELINES
KAFKA PRODUCER Producer > client毆戔䟖ṛރݺֵአොୗ (ܨᔴ坌懱௳ٚ蝑ڊ牧IM皤螛斃硵眤襑藲碉) Topic > Partition碍ᰁӤ褖ݢ犥ض懪2000 (憙ݱ蜃ᓕ๐率ಅ׀ጱӤ褖)
Partition > client毆戔ྯ稞䌃獈ฎ狅ଧ Partition 0→1→2 (憙襑穩藲碉౮ᓒkeyጱHash) 11
KAFKA CONSUMER Consumer ྯ稞襑ޞᎣ托奞ጱ襑穩 > Topic > Partition ID >
Offset Consumer Group ݱPartition虻碘ݝ蒂ቘӞ稞 > Topic 12
疩獈KAFKA 䌃獈猻ض蝱Kafka sync捝玲ᆙଉ蚎grpcො礯(䧙ਧ) Consumer > 朼䌘ݱUser牏ग़蕕ᗝ虻碘藶穩 Consumer Group > 朼䌘Worker狶瞙稞蒂ቘ
13
癲獈ٌਙᒫӣො๐率 MySQL > 㮆Ո牏ᗭ奲虻碘缛 Redis > 盠玲现夵袅瞂懿犥ے蝧疨ತ mongoDB > 瞱ԋ玕懱௳
14
ٌਙ襑穩 HTTP Proxy Server: Squid > K8S℄讨狅襑穩ᛔ㵕ी仂ֵ藶穩IP虋㵕 > ℂ襑ᒒࢧ苽瑿ᰂ窕๐率襑穩(ጮݷ㻌秚ګ) >
Develop, Staging, RC 絑ह缛㬵რ藶穩介手 15
ࢧ毣㬵፡፡K8Sঅ蒂 Service > 瞥獤๐率ኧService狶ፘ传蝢 犋ࣁNode樌ጱ翕᪠现 Podஂ種樌ই֜ਂ玲 Ingress > 蝚螂routingጱොୗ㬵ᓕቘ 襑ᥝ䌘क़ู襷ጱService
16
MIDDLEWARE ቘ薹 Proxy typical load-balancing > SSL termination > ኧReverse
Proxy Server究ਧፓ秂 17
SERVICE ܻቘ transparent tcp traffic proxy > Clientᒒ狒瞱ݶӞ礿TCP蝫娄 > 萞ኧፗ矑硬虋۱֖ᗝ螈౮
18
SERVICE HEADLESS DNS Client Easy solve L7 load-balancing > K8S
Service Headless玲Podڜ蔭 > Clientᒒ究ਧ揗斉ଘᤍ֖ᗝ 19
SERVICE MESH 禊盢Ӥݢ䋿֢ෝ覍瑊絑ह ᛔᤈᓕቘ虻碘窕牧犋አK8Sܻতො礯 ፓ獮๚䟖አService Meshෝ๐率Ӿ Data Plane > ݢ犥磦矦Proxy䋿֢
> 膏茴褸logፘݶ緡犥sidecar秇ୗ Control Plane > Proxy矒ګ > 狅窕ᰁ硈ਧᒽኼ > 蛪犩藨戣 20
CI / CD ਧ嬝窕纷膏䋿֢
JENKINSFILE ਧ嬝PR窕纷 > Prepare ୌ缏瑿絑ह㪔㬙ᗟᒫӣො๐率 (MySQL, Redis, mongoDB) 㪔蝢ᎣGitlab樄তPR窕纷牧纔ྊݳ㬫Branch >
Testing 䁆ᤈUnit Test㪔叨ኞ介手䁭ޞ > Build Image ଘᤈୌ缏ಅ磪盏๐率ጱimage > Deploy Local Service 蟂ᗟಅ磪盏๐率ᛗ瑿K8S Cluster > Report ݳ㬫ಅ磪褩ྦྷጱXML介手䁭ޞ㪔ԻᛗJenkins > Notification 咳蝑窕纷奾ຎᛗSlack 㪔蝢ᎣGitlab奾PR窕纷牧狅䁭ޞฎ玽ݶݳ㬫 22
JENKINSFILE ਧ嬝㬙ᗟ > Deploy Cloud Service 疥ض獮ಅ磪盏๐率ጱImage Ӥ㯽ᛗPrivate Registery 㪔蟂ᗟಅ磪盏๐率ᛗ
䌘䛑Branch絑हጱK8S Cluster > Notification 咳蝑窕纷奾ຎᛗSlack 㪔狅奾ຎฎ玽ᥝ䔶ګ蝢Ꭳ ࣁ娄ጱ姘螀Ո㹓ૡ纷䒍 (舙०硻K8S蛪䨝狶ࢧ笔) 23
NEXT? GitLab + Jira 碉ݳ 萞ኧ秂戢Jira Task翥蒈ᛗGitLab Jira疰胼ࣁGitLab獤ඪcommit, merge碻 ᛔ㵕疥Taskᑏᛗ蒂ቘӾ牏૪ਠ౮
ਠ珿ጱ㻌֖介手窩荠ሲ 犋㰍㰍ฎᥝ蝢螂Unit Test ইຎྯ稞ݳ㬫獮碉誢介手窩荠ሲ๚Ӥ܋ Ӟ䰬纔ྊPR狶ݳ㬫 Module Proxy Server ྯ稞叨ኞጱ獊碝介手絑ह 襑ᥝӞ㮆揗揣CacheᒫӣොॺկጱProxy Server slack bot 犋ٚ㰍㰍ݝฎ蝢Ꭳ 礍戔Bot Server ಋ㵕᯿珸๐率牏娄Ӥ碍硁翄懯奾ຎ 萞ኧ翄ӞࣁSlackՕᶎӤ砺֢㬵糷玲ݱୗ䁭ޞ 24
MORE CHALLENGE CRD(Custom Resource Definitions) > 疥ग़๐率ਧ嬝傶㻌Ӟ虻რ硯獈K8S or VM >
仂戔ਧ䲆ᓕቘ犖仂०藮 > 姘螀Ո㹓ๅ胼䌕ဳෝ碉誢๐率ጱᓕቘ 25
Q & A