Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How we build microservices with Kubernetes & Kafka
Search
Yun Chen
September 26, 2019
Programming
3
380
How we build microservices with Kubernetes & Kafka
An introduce to K8S and Kafka
Yun Chen
September 26, 2019
Tweet
Share
More Decks by Yun Chen
See All by Yun Chen
Redis Lua Script With Red Envelope & Message Queue
chenyunchen
0
600
Kubernetes Library with client-go
chenyunchen
0
410
[Graduate Project] Drupal Upgrade & Docker Deploy
chenyunchen
0
140
WECO Porject
chenyunchen
0
85
From Python Import Fun
chenyunchen
0
400
Docker
chenyunchen
2
190
NISRA fun with burp suite
chenyunchen
1
200
Vim
chenyunchen
0
160
Fun with programming
chenyunchen
0
160
Other Decks in Programming
See All in Programming
Compose 1.7のTextFieldはPOBox Plusで日本語変換できない
tomoya0x00
0
200
Jakarta EE meets AI
ivargrimstad
0
240
AWS Lambdaから始まった Serverlessの「熱」とキャリアパス / It started with AWS Lambda Serverless “fever” and career path
seike460
PRO
1
260
OSSで起業してもうすぐ10年 / Open Source Conference 2024 Shimane
furukawayasuto
0
110
色々なIaCツールを実際に触って比較してみる
iriikeita
0
330
Why Jakarta EE Matters to Spring - and Vice Versa
ivargrimstad
0
1.2k
最新TCAキャッチアップ
0si43
0
190
CSC509 Lecture 13
javiergs
PRO
0
110
Kaigi on Rails 2024 〜運営の裏側〜
krpk1900
1
240
みんなでプロポーザルを書いてみた
yuriko1211
0
280
3 Effective Rules for Using Signals in Angular
manfredsteyer
PRO
1
100
Functional Event Sourcing using Sekiban
tomohisa
0
100
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Become a Pro
speakerdeck
PRO
25
5k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
For a Future-Friendly Web
brad_frost
175
9.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
16
2.1k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Rails Girls Zürich Keynote
gr2m
94
13k
BBQ
matthewcrist
85
9.3k
Designing the Hi-DPI Web
ddemaree
280
34k
4 Signs Your Business is Dying
shpigford
180
21k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Practical Orchestrator
shlominoach
186
10k
Transcript
HOW WE BUILD MICROSERVICES WITH KUBERNETES & KAFKA SDN X
CLOUD NATIVE MEETUP #20 https://speakerdeck.com/chenyunchen/redis-lua-script-with-red-envelope-and-message-queue
WHO AM I Chen Yun Chen (Alex) >
[email protected]
>
blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
KUBERNETES ቘ薹硌獅膏褖ګ
CONTAINER ቘ薹膏褖ګ > Container != Docker > Appᥝெ讕瞥獤ग़㮆Dockerfile? > Log෭扮ইຎ茴褸?
> 緳矒᯿珸ᥝெ讕ڣ䥁? > 瑊絑ह牏܋羷ፘ狅䷱㺔氂? 4
KUBERNETES 硌獅膏褖ګ > Storage: CSI (Container Storage Interface) 吖瑿猋ൔ, 梊秚ګ,
盠ᆙ, ᯿蕦虻碘㳫ᴻ? > Network: CNI (Container Network Interface) ๅṛጱ窕ᰁ, ๅ犵ጱ皤螛, ๅग़ጱ翕᪠Օᶎ? > Computing: CRI (Container Runtime Interface) 犋አDocker? ೭㬵᪒VM? 窾ݳContainer&VM? 5
ܻত礍䯤 > 㻌ݣVM䁆ᤈ㮆誢 > ֵአᒫӣොጱRedis, MySQL๐率 6
疩獈K8S㪔瞥獤๐率 犥㻌Ӟ℄讨傶ֺ 繸ෝ缰ଏ牧蟂℄奞℄ኼ Auth Server > ض疥ጭ獈藨挨๐率ು櫝 Logic Server >
ݳဩጱ藶穩盅蝑狶蒂ቘ 7
疩獈WEBSOCKET蝫娄 Auth Server > 奈TCP蝫娄 ׀Android, iOS藨挨 WebSocket Server >
WebSocket蝫娄 ׀Web藨挨 翕殷粚氃क़螵蜉襑穩 8
瞥獤LOGIC๐率 Http Server > ׀獉蟂ٌਙ㿁褧 篷ဩ蝚螂gRPCොୗፗ蝫Logic Avatar Server > 蕣狶㮆Ո牏ᗭ奲瑽猟
Message Center Server > 懱௳๐率ፘ橕 File Server > 䲆礯๐率ፘ橕牏ݱ圵疳ੑ婘瑽蕣狶 9
KAFKA BUILDING REAL-TIME DATA PIPELINES
KAFKA PRODUCER Producer > client毆戔䟖ṛރݺֵአොୗ (ܨᔴ坌懱௳ٚ蝑ڊ牧IM皤螛斃硵眤襑藲碉) Topic > Partition碍ᰁӤ褖ݢ犥ض懪2000 (憙ݱ蜃ᓕ๐率ಅ׀ጱӤ褖)
Partition > client毆戔ྯ稞䌃獈ฎ狅ଧ Partition 0→1→2 (憙襑穩藲碉౮ᓒkeyጱHash) 11
KAFKA CONSUMER Consumer ྯ稞襑ޞᎣ托奞ጱ襑穩 > Topic > Partition ID >
Offset Consumer Group ݱPartition虻碘ݝ蒂ቘӞ稞 > Topic 12
疩獈KAFKA 䌃獈猻ض蝱Kafka sync捝玲ᆙଉ蚎grpcො礯(䧙ਧ) Consumer > 朼䌘ݱUser牏ग़蕕ᗝ虻碘藶穩 Consumer Group > 朼䌘Worker狶瞙稞蒂ቘ
13
癲獈ٌਙᒫӣො๐率 MySQL > 㮆Ո牏ᗭ奲虻碘缛 Redis > 盠玲现夵袅瞂懿犥ے蝧疨ತ mongoDB > 瞱ԋ玕懱௳
14
ٌਙ襑穩 HTTP Proxy Server: Squid > K8S℄讨狅襑穩ᛔ㵕ी仂ֵ藶穩IP虋㵕 > ℂ襑ᒒࢧ苽瑿ᰂ窕๐率襑穩(ጮݷ㻌秚ګ) >
Develop, Staging, RC 絑ह缛㬵რ藶穩介手 15
ࢧ毣㬵፡፡K8Sঅ蒂 Service > 瞥獤๐率ኧService狶ፘ传蝢 犋ࣁNode樌ጱ翕᪠现 Podஂ種樌ই֜ਂ玲 Ingress > 蝚螂routingጱොୗ㬵ᓕቘ 襑ᥝ䌘क़ู襷ጱService
16
MIDDLEWARE ቘ薹 Proxy typical load-balancing > SSL termination > ኧReverse
Proxy Server究ਧፓ秂 17
SERVICE ܻቘ transparent tcp traffic proxy > Clientᒒ狒瞱ݶӞ礿TCP蝫娄 > 萞ኧፗ矑硬虋۱֖ᗝ螈౮
18
SERVICE HEADLESS DNS Client Easy solve L7 load-balancing > K8S
Service Headless玲Podڜ蔭 > Clientᒒ究ਧ揗斉ଘᤍ֖ᗝ 19
SERVICE MESH 禊盢Ӥݢ䋿֢ෝ覍瑊絑ह ᛔᤈᓕቘ虻碘窕牧犋አK8Sܻতො礯 ፓ獮๚䟖አService Meshෝ๐率Ӿ Data Plane > ݢ犥磦矦Proxy䋿֢
> 膏茴褸logፘݶ緡犥sidecar秇ୗ Control Plane > Proxy矒ګ > 狅窕ᰁ硈ਧᒽኼ > 蛪犩藨戣 20
CI / CD ਧ嬝窕纷膏䋿֢
JENKINSFILE ਧ嬝PR窕纷 > Prepare ୌ缏瑿絑ह㪔㬙ᗟᒫӣො๐率 (MySQL, Redis, mongoDB) 㪔蝢ᎣGitlab樄তPR窕纷牧纔ྊݳ㬫Branch >
Testing 䁆ᤈUnit Test㪔叨ኞ介手䁭ޞ > Build Image ଘᤈୌ缏ಅ磪盏๐率ጱimage > Deploy Local Service 蟂ᗟಅ磪盏๐率ᛗ瑿K8S Cluster > Report ݳ㬫ಅ磪褩ྦྷጱXML介手䁭ޞ㪔ԻᛗJenkins > Notification 咳蝑窕纷奾ຎᛗSlack 㪔蝢ᎣGitlab奾PR窕纷牧狅䁭ޞฎ玽ݶݳ㬫 22
JENKINSFILE ਧ嬝㬙ᗟ > Deploy Cloud Service 疥ض獮ಅ磪盏๐率ጱImage Ӥ㯽ᛗPrivate Registery 㪔蟂ᗟಅ磪盏๐率ᛗ
䌘䛑Branch絑हጱK8S Cluster > Notification 咳蝑窕纷奾ຎᛗSlack 㪔狅奾ຎฎ玽ᥝ䔶ګ蝢Ꭳ ࣁ娄ጱ姘螀Ո㹓ૡ纷䒍 (舙०硻K8S蛪䨝狶ࢧ笔) 23
NEXT? GitLab + Jira 碉ݳ 萞ኧ秂戢Jira Task翥蒈ᛗGitLab Jira疰胼ࣁGitLab獤ඪcommit, merge碻 ᛔ㵕疥Taskᑏᛗ蒂ቘӾ牏૪ਠ౮
ਠ珿ጱ㻌֖介手窩荠ሲ 犋㰍㰍ฎᥝ蝢螂Unit Test ইຎྯ稞ݳ㬫獮碉誢介手窩荠ሲ๚Ӥ܋ Ӟ䰬纔ྊPR狶ݳ㬫 Module Proxy Server ྯ稞叨ኞጱ獊碝介手絑ह 襑ᥝӞ㮆揗揣CacheᒫӣොॺկጱProxy Server slack bot 犋ٚ㰍㰍ݝฎ蝢Ꭳ 礍戔Bot Server ಋ㵕᯿珸๐率牏娄Ӥ碍硁翄懯奾ຎ 萞ኧ翄ӞࣁSlackՕᶎӤ砺֢㬵糷玲ݱୗ䁭ޞ 24
MORE CHALLENGE CRD(Custom Resource Definitions) > 疥ग़๐率ਧ嬝傶㻌Ӟ虻რ硯獈K8S or VM >
仂戔ਧ䲆ᓕቘ犖仂०藮 > 姘螀Ո㹓ๅ胼䌕ဳෝ碉誢๐率ጱᓕቘ 25
Q & A