How we build microservices with Kubernetes & Kafka

An introduce to K8S and Kafka

Transcript

1. HOW WE BUILD MICROSERVICES WITH KUBERNETES & KAFKA SDN X CLOUD NATIVE MEETUP #20 https://speakerdeck.com/chenyunchen/redis-lua-script-with-red-envelope-and-message-queue
2. WHO AM I Chen Yun Chen (Alex) > me@yunchen.tw > blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
3. KUBERNETES ቘ薹硌獅膏褖ګ
4. CONTAINER ቘ薹膏褖ګ > Container != Docker > Appᥝெ讕瞥獤ग़㮆Dockerfile? > Log෭扮ইຎ茴褸? > 緳矒᯿珸ᥝெ讕ڣ䥁? > ਻瑊絑ह牏܋羷ፘ狅䷱㺔氂? 4
5. KUBERNETES 硌獅膏褖ګ > Storage: CSI (Container Storage Interface) 吖瑿猋ൔ, ਻梊秚ګ, 盠ᆙ, ᯿蕦虻碘㳫ᴻ? > Network: CNI (Container Network Interface) ๅṛጱ窕ᰁ, ๅ犵ጱ皤螛, ๅग़ጱ翕᪠Օᶎ? > Computing: CRI (Container Runtime Interface) 犋አDocker? ೭㬵᪒VM? 窾ݳContainer&VM? 5
6. ܻত礍䯤 > 㻌ݣVM䁆ᤈ㮆誢 > ֵአᒫӣොጱRedis, MySQL๐率 6
7. 疩獈K8S㪔瞥獤๐率 犥㻌Ӟ℄讨傶ֺ 繸ෝ缰ଏ牧蟂℄奞℄፜ኼ Auth Server > ض疥ጭ獈藨挨๐率ು櫝 Logic Server > ݳဩጱ藶穩಍盅蝑狶蒂ቘ 7
8. 疩獈WEBSOCKET蝫娄 Auth Server > 奈TCP蝫娄 ׀Android, iOS藨挨 WebSocket Server > WebSocket蝫娄 ׀Web藨挨 翕殷粚氃क़螵蜉襑穩 8
9. 瞥獤LOGIC๐率 Http Server > ׀獉蟂ٌਙ㿁褧 篷ဩ蝚螂gRPCොୗፗ蝫Logic Avatar Server > 蕣狶㮆Ո牏ᗭ奲瑽猟 Message Center Server > 懱௳๐率ፘ橕 File Server > 䲆礯๐率ፘ橕牏ݱ圵疳ੑ婘瑽蕣狶 9
10. KAFKA BUILDING REAL-TIME DATA PIPELINES
11. KAFKA PRODUCER Producer > client毆戔䟖൉ṛރݺֵአොୗ (ܨᔴ坌懱௳ٚ蝑ڊ牧IM皤螛斃硵眤襑藲碉) Topic > Partition碍ᰁӤ褖ݢ犥ض懪2000 (憙ݱ蜃ᓕ๐率ಅ൉׀ጱӤ褖) Partition > client毆戔ྯ稞䌃獈ฎ狅ଧ Partition 0→1→2 (憙襑穩藲碉౮ᓒkeyጱHash) 11
12. KAFKA CONSUMER Consumer ྯ稞襑ޞᎣ托奞ጱ襑穩 > Topic > Partition ID > Offset Consumer Group ݱPartition虻碘ݝ蒂ቘӞ稞 > Topic 12
13. 疩獈KAFKA 䌃獈猻ض蝱Kafka sync捝玲ᆙଉ蚎grpcො礯(䧙ਧ) Consumer > 朼䌘ݱUser牏ग़蕕ᗝ虻碘藶穩 Consumer Group > 朼䌘Worker狶瞙稞蒂ቘ 13
14. 癲獈ٌਙᒫӣො๐率 MySQL > 㮆Ո牏ᗭ奲虻碘缛 Redis > 盠玲现夵袅瞂懿犥ے蝧疨ತ mongoDB > 瞱ԋ玕懱௳ 14
15. ٌਙ襑穩 HTTP Proxy Server: Squid > K8S℄讨狅襑穩ᛔ㵕ी仂ֵ藶穩IP虋㵕 > ℂ襑ᒒ಑ࢧ苽瑿ᰂ窕๐率襑穩(ጮݷ㻌秚ګ) > Develop, Staging, RC 絑ह缛㬵რ藶穩介手 15
16. ࢧ毣㬵፡፡K8Sঅ蒂 Service > 瞥獤๐率ኧService狶԰ፘ传蝢 犋஠ࣁ఺᪜Node樌ጱ翕᪠现 Podஂ種樌ই֜ਂ玲 Ingress > 蝚螂routingጱොୗ㬵ᓕቘ 襑ᥝ䌘क़ู襷ጱService 16
17. MIDDLEWARE ቘ薹 Proxy typical load-balancing > SSL termination > ኧReverse Proxy Server究ਧፓ秂 17
18. SERVICE ܻቘ transparent tcp traffic proxy > Clientᒒ狒瞱ݶӞ礿TCP蝫娄 > 萞ኧፗ矑硬虋੗۱֖ᗝ螈౮ 18
19. SERVICE HEADLESS DNS Client Easy solve L7 load-balancing > K8S Service Headless玲஑Podڜ蔭 > Clientᒒ究ਧ揗斉ଘᤍ֖ᗝ 19
20. SERVICE MESH 禊盢Ӥݢ䋿֢ෝ覍਻瑊絑ह ᛔᤈᓕቘ虻碘窕牧犋አK8Sܻতො礯 ፓ獮੢๚䟖አService Meshෝ๐率Ӿ Data Plane > ݢ犥磦矦Proxy䋿֢ > 膏茴褸logፘݶ緡犥sidecar秇ୗ Control Plane > Proxy矒ګ > 狅窕ᰁ硈ਧᒽኼ > 蛪犩藨戣 20
21. CI / CD ਧ嬝窕纷膏䋿֢
22. JENKINSFILE ਧ嬝PR窕纷 > Prepare ୌ缏๜瑿絑ह㪔㬙ᗟᒫӣො๐率 (MySQL, Redis, mongoDB) 㪔蝢ᎣGitlab樄তPR窕纷牧纔ྊݳ㬫Branch > Testing 䁆ᤈUnit Test㪔叨ኞ介手䁭ޞ > Build Image ଘᤈୌ缏ಅ磪盏๐率ጱimage > Deploy Local Service 蟂ᗟಅ磪盏๐率ᛗ๜瑿K8S Cluster > Report ݳ㬫ಅ磪褩ྦྷጱXML介手䁭ޞ㪔൉ԻᛗJenkins > Notification 咳蝑窕纷奾ຎᛗSlack 㪔蝢ᎣGitlab奾๳PR窕纷牧狅䁭ޞฎ玽ݶ఺ݳ㬫 22
23. JENKINSFILE ਧ嬝㬙ᗟ > Deploy Cloud Service 疥ض獮ಅ磪盏๐率ጱImage Ӥ㯽ᛗPrivate Registery 㪔蟂ᗟಅ磪盏๐率ᛗ 䌘䛑Branch絑हጱK8S Cluster > Notification 咳蝑窕纷奾ຎᛗSlack 㪔狅奾ຎฎ玽ᥝ䔶ګ蝢Ꭳ ࣁ娄ጱ姘螀Ո㹓౲ૡ纷䒍 (舙०硻K8S๜蛪䨝狶ࢧ笔) 23
24. NEXT? GitLab + Jira 碉ݳ 萞ኧ秂戢Jira Task翥蒈ᛗGitLab Jira疰胼ࣁGitLab獤ඪcommit, merge碻 ᛔ㵕疥Taskᑏᛗ蒂ቘӾ牏૪ਠ౮ ਠ珿ጱ㻌֖介手窩荠ሲ 犋㰍㰍ฎᥝ蝢螂Unit Test ইຎྯ稞ݳ㬫獮碉誢介手窩荠ሲ๚Ӥ܋ Ӟ䰬纔ྊPR狶ݳ㬫 Module Proxy Server ྯ稞叨ኞጱ獊碝介手絑ह 襑ᥝӞ㮆揗揣CacheᒫӣොॺկጱProxy Server slack bot 犋ٚ㰍㰍ݝฎ蝢Ꭳ 礍戔Bot Server ಋ㵕᯿珸๐率牏娄Ӥ碍硁翄懯奾ຎ 萞ኧ翄ӞࣁSlackՕᶎӤ砺֢㬵糷玲ݱୗ䁭ޞ 24
25. MORE CHALLENGE CRD(Custom Resource Definitions) > 疥ग़๐率ਧ嬝傶㻌Ӟ虻რ硯獈K8S or VM > 仂੝戔ਧ䲆ᓕቘ犖仂੝०藮 > 姘螀Ո㹓ๅ胼䌕ဳෝ碉誢๐率ጱᓕቘ 25
26. Q & A