Kubernetes 運用設計ガイド / A design guide for Kubernetes in production (Japanese)

2018/04/19 JAPAN CONTAINER DAYS V18.04 (https://containerdays.jp/) にて発表したものを加筆修正しました。

Abstract: Kubernetes は豊富な機能とその高い拡張性により、現実における様々なユースケースに対応できる一方、その多機能さゆえにどう使えば良いか迷っている方もいると思います。Kubernetes の基本を学んだ人や本番運用を始めた人を対象に、私がメルカリでの Kubernetes 本番運用経験を元に考えた、アプリケーション運用、インフラ運用、組織の 3 つの観点での設計の指針を紹介します。

Transcript

1. Kubernetes ӡ༻ઃܭΨΠυ Japan Container Days v1804, Apr 19, 2018
2. @spesnova SRE at Mercari,Inc. / Kubernetes tokyo community organizer
3. ࠓ೔ͷςʔϚ ˞
   ͜͜Ͱड़΂Δ಺༰͸ॴଐ૊৫ͷެࣜݟղͰ͸ͳ͘ݸਓͷݟղͰ͢
4. Kubernetes ΛͲ͏࢖ͬͨΒ͍͍͔໎ͬͯΔํ λʔήοτ
5. Kubernetes Λ࢖͏໨తΛ໌֬ʹ͠Α͏ ໌֬ͳ໨తΛ࣋ͬͯར༻͢Ε͹ɺ ɹͲ͏࢖͑͹ྑ͍͔ࣗવʹݟ͑ͯ͘Δ
6. ໌֬ͳ໨త(ઃܭํ਑)Λϕʔεʹ Kubernetes Λ Ͳ͏࢖͏͔ઃܭ͍ͯ͘͠ྫΛࣔ͢ ςʔϚ
7. ࣗ཯తͳνʔϜͱγεςϜΛ࡞Δ ໨త (ઃܭํ਑)
8. ࣗ཯తͱ͸ ͦͷ΋ͷࣗମ͚ͩͰௐ੔ΛߦͬͨΓɺ ໰୊Λղܾͨ͠ΓͳͲΛߦ͏͞· Ҿ༻
   IUUQTXXXXFCMJPKQDPOUFOUࣗ཯త
9. ͳͥࣗ཯త͔ʁ ྑ͍ϓϩμΫτΛ࡞ΔͨΊͷॏཁͳཁૉͩͱߟ͍͑ͯΔ͔Βɻ ࣗ཯తͳνʔϜ΍γεςϜ͸ͦ͏Ͱͳ͍৔߹ͱൺֱͯ͠ɺΑΓ଎͘ಈ ͘͜ͱ͕ՄೳͰɺΠϊϕʔγϣϯ΋ى͖΍͍͢ͱײ͍ͯ͡Δɻ
10. ࣗ཯తͳνʔϜͷྫ: ΞϝϦΧ܉ ਆग़َ຅ͳςϩ΍ήϦϥΛલʹͯ͠ɺΞϝϦΧ܉͸ϐϥϛουܕ૊৫ʹݶ քΛײ͡ɺωοτϫʔΫܕ૊৫ʹҠߦɻ্૚෦ʹҙࢥܾఆΛڼ͕ͣɺ࠷લ ઢ෦ୂ͕ࣗ਎Ͱҙࢥܾఆͯ͠ಈ͘͜ͱͰςϩͷεϐʔυײʹରԠͨ͠ɻ ࢀߟ
    5&".
    0'
    5&".4
    
11. ςϩ 1. ςϩΛݕ஌ 3. ҙࢥܾఆ 2. ্૚෦΁఻ୡ 5. ߦಈ 4. ݱ৔΁఻ୡ ϐϥϛουܕɺதԝूݖܕͷҙࢥܾఆ ࢀߟ
    5&".
    0'
    5&".4
    
12. ςϩ 3. ߦಈͱ৘ใڞ༗ ωοτϫʔΫܕɺ෼ࢄܕͷҙࢥܾఆ 2. ҙࢥܾఆ ࢀߟ
    5&".
    0'
    5&".4
    1. ςϩΛݕ஌
13. ࣗ཯తͳγεςϜͷྫ: Netflix AWS EC2 ͷϝϯςφϯεϦϒʔτΛࣗಈ෮چͷ࢓૊ΈʹΑͬͯθϩ μ΢ϯλΠϜͰ৐Γ੾ͬͨɻ͜Ε͸Ϧϒʔτ͕͔͔Δ౓ʹ OnCall ୲ ౰ऀʹి࿩͕໐ΓɺखಈͰ෮چ͢ΔΑΓ΋଎͍ɻ ࢀߟ
    IUUQTXXXTMJEFTIBSFOFUQMBOFUDBTTBOESBOFUqJYBTUBUFPGYFODIBPTNPOLFZDBTTBOESB
14. ଎͞ͱ͍͏ͷ͸ٸ͙͜ͱ͔ΒͰ͸ͳ͘ɺ Կ͔Λແ͘͢͜ͱ͔Βੜ·ΕΔ ଎͘ಈ͘
15. Կ͔Λແͯ͘͠଎͘ͳͬͨྫ • ΞϝϦΧ܉ͷྫ: ্૚෦ͷҙࢥܾఆ଴ͪΛແ͘͢ • Netflix ͷྫ: ਓ΁ͷ࿈བྷͱख࡞ۀΛແ͘͢ • αʔόͷϨεϙϯελΠϜ: DB ΁ͷ໰͍߹ΘͤΛݮΒ͢ • Amazon ͓ٸ͗ศ: ૔ݿΛ૿΍ͯ͠༌ૹڑ཭ΛݮΒ͢
16. Kubernetes ࣗମͷૂ͍ͱζϨ͍ͯͳ͍͔ʁ Kubernetes is more than just a “container orchestrator”. It aims to eliminate the burden of orchestrating physical/ virtual compute, network, and storage infrastructure, and enable application operators and developers to focus entirely on container-centric primitives for self-service operation. Kubernetes Design and Architecutre Ҿ༻
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE
17. Kubernetes ࣗମͷૂ͍ͱζϨ͍ͯͳ͍͔ʁ 1. Portable 2. General-Purpose 3. Meet users partway 4. Flexible 5. Extensible 6. Automatable 7. Advance the state of the art Kubernetes Design and Architecutre Ҿ༻
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE
18. Kubernetes ࣗମͷૂ͍ Ҿ༻
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE 1. Πϯϑϥ؅ཧͷͨΊͷख࡞ۀΛܶతʹݮΒ͢͜ͱ 2. ηϧϑαʔϏεܕͷӡ༻ΛՄೳʹ͢Δ͜ͱ
19. ࣗ཯తͳνʔϜͱγεςϜΛ࡞ΔͨΊʹ 1. γεςϜͷந৅Խɺૄ݁߹Խɺ࡞ۀͷࣗಈԽ 2. ڞ௨໨తͷ໌֬Խɺద੾ͳ੹຿ͷ෼ղͱఆٛɺݖݶҕৡ
20. νʔϜͷઃܭ
21. ͳͥνʔϜͷઃܭ͔Βʁ 1. ٕज़ͱ૊৫͸දཪҰମ 2. Kubernetes ͸ਓͷͨΊʹɺਓʹΑͬͯ࡞ΒΕɺਓʹΑͬ ͯӡ༻͞ΕΔɺ࢖͏ଆ΋ਓʹ͍ͭͯߟ͑Δඞཁ͕͋Δ 3. ૊৫จԽͱΠϯϑϥΛ྆ྠͱͯ͠ಉ࣌ʹม͍͑ͯͬͯॳ ΊͯޮՌ͕ग़Δ
22. “૊৫ͷઃܭ͢ΔγεςϜ͸ɺͦͷ૊৫ͷ ɹɹίϛϡχέʔγϣϯߏ଄Λͦͷ··൓өͨ͠ઃܭʹͳΔ” ίϯ΢ΣΠͷ๏ଇ
23. ։ൃڌ఺͕ 3 ͭ͋Δͱιʔείʔυ΋ 3 ͭʹ෼͔ΕΔ ։ൃڌ఺ ιʔείʔυ
24. ։ൃڌ఺͕ 3 ͭ͋Δͱιʔείʔυ΋ 3 ͭʹ෼͔ΕΔ ։ൃڌ఺ ιʔείʔυ
25. ։ൃڌ఺͕ 1 ͭͩͱւ֎ల։ͯ͠΋ιʔείʔυ͸ 1 ͭ ։ൃڌ఺ ιʔείʔυ αʔϏε
26. “࡞Γ͍ͨγεςϜͷߏ଄Λ൓өͨ͠ίϛϡχέʔγϣϯ͓Αͼ ɹɹ૊৫ߏ଄Λ·ͣ࡞ΔͱɺγεςϜ͕ظ଴ͨ͠ઃܭʹͳΔ” ίϯ΢ΣΠͷ๏ଇΛٯखʹऔΔ
27. Kubernetes ͷੈքͰ͸γεςϜ͸ 2 छྨʹେผͰ͖Δ Πϯϑϥڞ௨ج൫ܥ ΞϓϦέʔγϣϯܥ
28. Πϯϑϥڞ௨ج൫ܥ ΞϓϦέʔγϣϯܥ ΞϓϦέʔγϣϯͱڞ௨ج൫Λີ݁߹ͤͯ͞͸͍͚ͳ͍
29. 2 छྨͷνʔϜΛ࡞Δ ΫϥελΞυϛϯνʔϜ ϓϩμΫτνʔϜ
30. ੹೚ൣғͷઃܭ
31. γεςϜ͝ͱʹඞཁͳΤϯδχΞϦϯά࡞ۀ͕͋Δ Πϯϑϥڞ௨ج൫ܥ ΞϓϦέʔγϣϯܥ
32. • Ϋϥελͷϝϯςφϯε • ڞ௨ίϯϙʔωϯτͷϝϯςφϯε • σϓϩΠύΠϓϥΠϯ • ΫϥελϨϕϧͷϞχλϦϯά • ΫϥελϨϕϧͷηΩϡϦςΟ etc • ΞϓϦέʔγϣϯίʔυ • ςετ • ίϯςφԽ • σϓϩΠ • ΞϓϦέʔγϣϯϨϕϧͷϞχλϦϯά etc ΞϓϦέʔγϣϯܥ Πϯϑϥڞ௨ج൫ܥ γεςϜ͝ͱʹඞཁͳΤϯδχΞϦϯά࡞ۀ͕͋Δ
33. ࣗ཯తʹಈͨ͘Ίɺૄ݁߹ʹͳΔ੹೚ڥքΛܾΊΔ Ϋϥελ ίϯςφ ϊʔυ ιʔείʔυ
34. ϓϩμΫτνʔϜͷ੹೚ൣғ ސ٬ͷ՝୊Λղܾ͢Δ͜ͱɻͦͷͨΊʹϓϩμΫτͷاը͔Β σβΠϯɺ։ൃɺӡ༻·Ͱશͯʹ੹೚Λ࣋ͭɻ
35. ΫϥελΞυϛϯνʔϜͷ੹೚ൣғ ϓϩμΫτνʔϜͷύϑΥʔϚϯεΛ࠷େԽ͢Δ͜ͱɻͦͷͨ ΊʹɺσϓϩΠύΠϓϥΠϯ΍ϞχλϦϯάͳͲͷϓϩμΫτ νʔϜΛࢧ͑Δڞ௨ج൫ͷ։ൃ͔Βӡ༻·Ͱ੹೚Λ࣋ͭɻ
36. “You build it, you run it”
37. Ϋϥελͷઃܭ
38. ։ൃ؀ڥɺຊ൪؀ڥ͝ͱʹΫϥελΛ࡞Δʁ Development Ϋϥελ Ϋϥελ Production
39. 1. ࠷΋Ұൠతͳཻ౓ 2. εςʔδϯά؀ڥ͕ඞཁʹͳͬͨΒʁ QA ؀ڥ͸ʁ 3. ؀ڥ͕૿͑ΔͨͼʹΫϥελ͕૿͑ͯ؅ཧίετ্͕͕Δ 4. ։ൃ؀ڥͱຊ൪؀ڥ͕Ұகͯ͠Δ͜ͱ͕อূͮ͠Β͍ ։ൃ؀ڥɺຊ൪؀ڥ͝ͱʹΫϥελΛ࡞Δʁ
40. Ϧʔδϣϯ͝ͱʹ 1 ͚ͭͩΫϥελΛ࡞Δ London Ϋϥελ Tokyo Ϋϥελ California Ϋϥελ
41. Ϧʔδϣϯ͝ͱʹ 1 ͚ͭͩΫϥελΛ࡞Δ London Ϋϥελ Development Production Staging BranchLab QA Sandbox etc
42. Ϧʔδϣϯ͝ͱʹ 1 ͚ͭͩΫϥελΛ࡞Δ 1. ͋ΒΏΔ؀ڥΛड͚ೖΕΒΕΔ 2. ։ൃ؀ڥͱຊ൪؀ڥ͕Ұகͯ͠Δ͜ͱ͕อূ͠΍͍͢ 3. ϓϩμΫτνʔϜ͕ΫϥελΛҙࣝ͠ͳ͍͍ͯ͘ (ந৅Խ) 4. ։ൃ؀ڥ͕ຊ൪؀ڥʹѱӨڹΛ༩͑ͳ͍͔ෆ҆ʁ
43. ؀ڥΛಛผࢹ͠ͳ͍ Service A Development Service B Production Service A Production Service B Production ։ൃ؀ڥͱຊ൪؀ڥΛ෼͚ͨͱͯ͠΋ɺ͋ΔαʔϏε͕͋ΔαʔϏεʹӨڹΛ༩͑ ͳ͍Α͏ʹ͠ͳ͍ͱ͍͚ͳ͍͜ͱʹ͸มΘΓ͸ͳ͍ɻͩͱ͢Ε͹։ൃ؀ڥͱຊ൪ ؀ڥΛࠞࡏͤͯ͞΋ಉ͡Ͱ͋Δ
44. Ϧʔδϣϯ͝ͱʹ 1 ͚ͭͩΫϥελΛ࡞Δ 1. AWS, GCP, Heroku ʹ։ൃ؀ڥઐ༻૭ޱ͸ͳ͍ 2. Ϣʔβʔ͕”։ൃ؀ڥ༻”ͱͯ͠ΞΧ΢ϯτΛ࡞͍ͬͯΔ͚ͩ 3. Google ΋ GitHub ΋ Cluster per region Ͱ͋Δ 4. ηΩϡΞͳαʔϏε΋ಉ͡ΫϥελʹೖΕΔͷ͔ʁ
45. ϓϩμΫτɺαʔϏε͝ͱʹΫϥελΛ෼͚Δʁ 1. ؀ڥ͝ͱʹΫϥελΛ࡞ΔΑΓ΋Ϋϥελ͕૿͑ͯ͠·͏ 2. ৴པͰ͖ͳ͍ୈࡾऀʹར༻ͯ͠΋Β͏౳ͷέʔε͸෼͚Δ΂͖ 3. ࣾ಺ʹ͋ΔଞͷαʔϏεͱશ͘௨৴͠ͳ͍ͳΒ͹༗Γ 4. ࣾ಺ʹ͋ΔଞͷαʔϏεͱશ͘௨৴͢Δ৔߹͸ʁ
46. ϓϩμΫτɺαʔϏε͝ͱʹΫϥελΛ෼͚Δʁ Secure Ϋϥελؒ௨৴ Service A Ϋϥελ಺௨৴ Secure Service A Default Network Policy ΍ Istio ͳͲͷଘࡏʹΑͬͯݱ࣌఺Ͱ͸Ϋϥελ಺௨৴ͷํ͕ωο τϫʔΫͷ੍ޚ͕͠΍͍͢ɻ
47. ಉ͡ϊʔυʹࡌ͍ͬͯΔ͜ͱ͕ґવͱͯ͠໰୊ Secure Ϋϥελ ϊʔυ Service A ίϯςφ
48. Ϋϥελ͸ 1ͭͷ··Ͱઐ༻ϊʔυΛ༻ҙ͢Δ Ϋϥελ ϊʔυ Service A Secure
49. ηΩϡΞʹ͍ͨ͠ίϯςφΛઐ༻ϊʔυʹ഑ஔ Ϋϥελ ϊʔυ Service A Secure
50. Ϋϥελͷઃܭ·ͱΊ 1. Ϧʔδϣϯ͝ͱʹ 1 ͚ͭͩΫϥελΛ࡞Δ 2. ؀ڥ͸Ϋϥελ಺෦ͷ isolation ٕज़ʹΑͬͯ෼཭͢Δ 3. ಛఆͷαʔϏεઐ༻ϊʔυ͸ຊ౰ʹඞཁͳ࣌ʹ͚ͩ༻ҙ͢Δ 4. ϓϩμΫτ/αʔϏε͝ͱʹΫϥελΛ࡞Δͷ͸࠷ޙͷखஈ
51. Namespace ͷઃܭ
52. Namespace ͰόʔνϟϧΫϥελ͕࡞੒Ͱ͖Δ Ϋϥελ όʔνϟϧΫϥελ
53. 1 ͭͷΫϥελͰ͋ΒΏΔ؀ڥΛड͚ೖΕΔ London Ϋϥελ Development Production Staging BranchLab QA Sandbox etc
54. ؀ڥ͝ͱʹ Namespace Λ෼͚Δ dev prod qa
55. Service A Development Service B Production Service A Production Service B Production ؀ڥ͚ͩͰͳ͘αʔϏε΋෼཭͍ͨ͠
56. αʔϏε໊+؀ڥ͝ͱʹ Namespace Λ෼͚Δ A-dev A-prod B-qa
57. Network Policy ͷઃܭ
58. Service A Development Service B Production Service A Production Service B Production Network Policy Ͱ Pod ؒͷ௨৴੍͕ޚͰ͖Δ
59. Namespace ϨϕϧͰ੍ޚ͢Δ A-prod ϙϦγʔ͕ෳࡶʹͳΓա͗ΔͷΛ๷͙ɻجຊతͳར༻έʔε͸αʔϏεؒͷ௨৴ Λ੍ޚͳͷͰɺαʔϏε͝ͱʹ Namespace Λ࡞͍ͬͯΔͷ͕׆͖ͯ͘Δɻ B-prod
60. جຊ͸ All Deny B-prod A-prod C-qa B-dev
61. ϗϫΠτϦετͰ௨৴Մೳͳ Namespace Λࢦఆ A-prod B-prod B-dev C-qa D-prod
62. RBAC ͷઃܭ
63. RBAC Ͱ Kubernetes ͷݖݶͷ؅ཧ͕Ͱ͖Δ • Deployment ͷ࡞੒ • Secrets ͷӾཡ • PVC ͷ࡟আ ϢʔβʔΞΧ΢ϯτ foo Role RoleBinding
64. RBAC Λݖݶҕৡʹར༻͢Δ Ϋϥελ Namespace “admin” Role ϓϩμΫτνʔϜ A Namespace A ΫϥελΞυϛϯνʔϜ “custom-cluster-admin” Role
65. Namespace Admin Role 1. ϓϦηοτͷ “admin” ΛϓϩμΫτνʔϜʹ෇༩ 2. ಛఆͷ Namespace ҎԼͷ؅ཧݖݶ ΛϓϩμΫτνʔϜʹݖݶҕৡ 3. ϓϩμΫτνʔϜଆͰඞཁʹԠͯ͡ edit(read-write) ΍ view(read-only) Λ࡞੒ 4. ૊৫తʹ੹೚ൣғΛ໌֬ʹఆٛ͢Δ͜ͱͱɺγεςϜతʹͦΕΛදݱ͢Δ͜ͱ ͸྆ྠɺͲͪΒ͕͚ܽͯ΋͍͚ͳ͍
66. Custom Cluster Admin Role 1. ϓϦηοτͷ “cluster-admin” ͸ԿͰ΋ग़དྷͯ͠·͏ 2. ݖݶΛ࣋ͭ͜ͱ͸ಉ࣌ʹ੹೚Λ࣋ͭ͜ͱΛҙຯ͢Δ 3. “cluster-admin” ͔ΒݖݶΛམͱͨ͠ “custom-cluster-admin” Λ༻ҙ 4. ڞ௨ج൫ʹར༻͢Δ namespace ΍ node ؅ཧʹඞཁͳݖݶΛ෇༩ 5. αʔϏε/ϓϩμΫτ༻ namespace ͸ secrets Λআ͖ view ݖݶΛ෇༩ 6. ͋͘·ͰΫϥελͷ؅ཧʹప͠ɺϓϩμΫτʹؔ͢Δ෦෼͸೚ͤΔ 7. ϓϩμΫτͷ৴པੑͷ୲อ͕ඞཁͳ৔߹ɺSRE ͸ΫϥελΞυϛϯͰ ͸ͳ͘ϓϩμΫτνʔϜʹॴଐͯ͠׆ಈ͢Δ
67. ΞϓϦέʔγϣϯίϯςφͷઃܭ
68. (Deployment / Service ౳ͷઃܭ)
69. 1. ࣗ཯తͳΞϓϦέʔγϣϯίϯςφΛ໨ࢦ͢ 2. ίϯςφͷҟৗऴྃɺϊʔυো֐΍ϊʔυϝϯςφϯε࣌ʹඋ͑ͯࣗಈ෮چ 3. ෛՙͷ૿ݮʹඋ͑ͯࣗಈεέʔϧ 4. ͱ͸͍͑׬શʹશࣗಈ͸೉͍͠ͷͰ࣮֬ʹखಈ࡞ۀ͸ඞཁʹͳΔ 5. खಈ࡞ۀָ͕ʹͳΔΑ͏ʹ΋ߟྀ ΞϓϦέʔγϣϯίϯςφͷઃܭ
70. 1. Observable: ίϯςφ͕ਖ਼ৗ͔ҟৗ͔൑ผ͕͚ͭΒΕΔ͔ɺ໰୊ൃੜ࣌ʹݪҼڀ໌͕Ͱ͖Δ͔ 2. Disposable: ҟৗऴྃͨ͠ίϯςφ΍ো֐தͷϊʔυ্ʹ͋ΔίϯςφΛ͙͢ʹࣺͯΒΕΔ͔ 3. Immutable: ϩʔϧόοΫ΍εέʔϧ࣌ʹಉ͡ίϯςφ͕ىಈ͢Δ͜ͱΛอূͰ͖Δ͔ 4. Scalable: ϩʔυςετΛܦͯεέʔϧͷ͖͍͠஋͕ఆ·͍ͬͯΔ͔ 5. Loosely Coupled: σϓϩΠɺϩʔϧόοΫɺεέʔϧ࣌ʹґଘؔ܎Λߟྀ͠ͳͯ͘ࡁΉ͔ 6. Graceful: ѱӨڹΛग़ͣ͞ʹىಈɺఀࢭ͕Ͱ͖Δ͔ ࣗಈ / खಈ෮چɺࣗಈ / खಈεέʔϧʹඞཁͳཁૉ
71. 1. Liveness Probe ͷར༻ 2. Readiness Probe ͷར༻ 3. ϩά ͷऩू 4. ϝτϦΫεͷऩू 5. τϨʔγϯά ᶃ Observable ϩά ϝτϦΫε τϨʔε ϔϧενΣοΫ ίϯςφ
72. Liveness Probe Liveness Probe Ͱ͸ϔϧενΣοΫʹ௨Βͳ͍৔߹ Kubernetes ͕ Pod (ίϯς φ) Λ࠶ىಈ͢ΔɻͦͷͨΊɺΞϓϦέʔγϣϯ͕ਖ਼ৗʹىಈ͔ͨ͠Λ൑ผ͢Δͨ Ίʹར༻͢Δɻٯʹݴ͏ͱɺͲ͏͍͏ঢ়گͰࣗಈ࠶ىಈ͍͔ͤͨ͞Λදݱ͢Δ৔ ॴͰ͋ΓɺKubernetes ʹඋΘ͍ͬͯΔࣗಈ෮چͷ࢓૊ΈͷҰ෦Ͱ͋Δɻϔϧε νΣοΫͷਫ਼౓͕؁͍ͱෆඞཁʹ࠶ىಈͯ͠͠·͏ॾਕͷ݋ͳ໘΋͋Δɻ
73. Liveness Probe ΞϓϦέʔγϣϯίϯςφ ϔϧενΣοΫ ΞϓϦέʔγϣϯίϯςφ ϔϧενΣοΫ New!
74. Readiness Probe Readiness Probe Ͱ͸ϔϧενΣοΫʹ௨ͬͨ৔߹ Kubernetes ͕ͦͷ Pod(ίϯ ςφ)Λ Service (ϩʔυόϥϯα) ʹొ࿥͢ΔɻͦͷͨΊɺσʔλϕʔε઀ଓͳͲ ΋ؚΊͯΞϓϦέʔγϣϯ͕ਖ਼ৗʹϨεϙϯεΛฦͤΔঢ়ଶ = ४උ͕Ͱ͖͔ͨ (Ready)Λ൑ผ͢ΔͨΊʹར༻͢Δɻ
75. Readiness Probe ΞϓϦέʔγϣϯίϯςφ ϔϧενΣοΫ ΞϓϦέʔγϣϯίϯςφ ϔϧενΣοΫ σʔλϕʔε σʔλϕʔε Service(ϩʔυόϥϯα) Service(ϩʔυόϥϯα)
76. 1. εςʔτϨεʹ͢Δ 2. σʔλ͸ӬଓԽετϨʔδʹ 3. ϩά͸ JSON ܗࣜͰඪ४ग़ྗʹ ᶄ Disposable ίϯςφ ϩά DB σʔλ ඪ४ग़ྗ
77. 1. Latest tag ͸࢖Θͳ͍ (ྫ: hello:1.0.1) 2. ։ൃ؀ڥͱຊ൪؀ڥͰಉ͡ΠϝʔδΛ࢖͏ ᶅ Immutable hello:1.0.1 ։ൃ؀ڥ hello:1.0.1 ຊ൪؀ڥ
78. 1. ϦϦʔεલʹϩʔυςετΛߦ͍εέʔϧͷ͖͍͠஋Λग़͓ͯ͘͠ 2. Horizontal Pod Autoscaler Λ࢖͏ 3. (Vertical Pod Autoscaler Λ࢖͏) 4. Pod Disruption Budget Λ࢖͏ 5. Pod Priority ᶆ Scalable
79. Horizontal Pod Autoscaler ίϯςφ CPU: 1 Memory: 1GB ίϯςφͷ਺: 3 ίϯςφͷ਺: 3 + 6
80. Vertical Pod Autoscaler ίϯςφ CPU: 1 Memory: 1GB ίϯςφ CPU: 1 Memory: 2GB ίϯςφ CPU: 1 Memory: 1GB ίϯςφ CPU: 2 Memory: 2GB
81. Pod Disruption Budget ίϯςφͷ਺: 10 PDB ͷྫ: ϊʔυϝϯςφϯε౳ͷܭըతͳϊʔυͷμ΢ϯ λΠϜ࣌ʹશମͷ 20% ·Ͱ͔͠ݮΔ͜ͱΛڐ༰͠ͳ͍ Քಇ͍ͯ͠Δίϯςφͷ਺: 8 ఀࢭͨ͠ίϯςφͷ਺: 2
82. Pod Disruption Budget ΫϥελΞυϛϯνʔϜ͕ɺܭըϝϯςφϯεʹΑͬͯϊʔυ͔Βίϯς φΛୀආͤ͞Α͏ͱͨ͠ͱ͖(kubectl drain)ɺKubernetes ͸ PDB ʹࢦఆ ͞Εͨ Pod ਺ΛԼճΒͳ͍Α͏ʹ͠ͳ͕ΒίϯςφΛୀආͤͯ͘͞ΕΔɻ ͜Ε͸ϓϩμΫτνʔϜͱΫϥελΞυϛϯνʔϜ͕͓ޓ͍ʹӨڹΛग़͞ ͳ͍Α͏ʹ͢ΔͨΊͷ࢓૊Έ = ૄ݁߹ԽͰ͋Δ
83. 1. 1 ͭͷίϯςφʹ͸ 1 ͭͷ࢓ࣄΛͤ͞Δ 2. ϋʔυίʔυ΍ґଘؔ܎Λۃྗආ͚Δ 1. Label ͷར༻ 2. σϓϩΠॱং͸͋ΔΑΓͳ͍ํ͕͍͍ 3. Node Affinity ΋ۃྗආ͚Δ 4. Service Λ࢖ͬͯݻఆ IP ΋ආ͚Δ ᶇ Loosely Coupled
84. ᶇ Loosely Coupled ಛఆͷ؀ڥԼͰ͔͠ੜ͖ΒΕͳ͍ੜ෺ΑΓ΋ɺ͋ΒΏΔ؀ڥԼͰੜ͖ΒΕ Δੜ෺ͷํ͕αόΠόϧೳྗ͕ߴ͍ͱݴ͑Δɻಉ༷ʹಛఆͷ໊લɺಛఆͷ ॱংɺಛఆͷϊʔυɺಛఆͷ IP ʹґଘͨ͠ίϯςφΑΓ΋ɺ ґଘ͕ͳ͍ίϯςφͷํ͕৴པੑ͕ߴ͍ɻ৴པੑ͕ߴ͍ίϯςφ͸ͦͷ෼ ख͕͔͔Βͳͯ͘ྑ͍ɻ
85. 1. 1 ίϯςφ 1 ϓϩηε͕جຊ 2. 1 ͭͷίϯςφ(Πϝʔδ)ʹෳ਺ͷ੹຿͕͋Δͱίϯςφͷྑ͕͞ͳ͘ͳΔ 1. Dockerfile ͕ෳࡶʹͳΔ 2. εέʔϧ৚͕݅ෳࡶʹͳΔ 3. ϞχλϦϯά͕ෳࡶʹͳΔ 4. ىಈॲཧɺఀࢭॲཧ͕ෳࡶʹͳΔ 1 ͭͷίϯςφʹ͸ 1 ͭͷ࢓ࣄΛͤ͞Δ
86. 1. SIGTERM, SIGKILL ΛϋϯυϦϯάͰ͖ΔΑ͏ʹ 2. ѱӨڹΛग़ͣ͞ʹఀࢭͰ͖ΔΑ͏ʹ 3. جຊతʹ͸࢖Θͳ͍ํ͕ෳࡶʹͳΒͣʹ͍͍͕ඞཁͳΒ͹ىಈޙ ͷॲཧɺఀࢭલͷॲཧʹ preStop, postStart ϋϯυϥΛ࢖͏ ᶈ Graceful
87. 12 Factor App
88. ΦϖϨʔγϣϯͷઃܭ
89. 1. ͜͜Ͱ͍͏ΦϖϨʔγϣϯͱ͸ʁ 1. ίϯςφͷσϓϩΠ(ྫ: Deployment ͷ࡞੒) 2. ڞ௨ج൫ͷϝϯςφϯε(ྫ: Node ͷΞοϓάϨʔυ) 2. ۃྗࣗ཯తͳγεςϜʹ೚ͤΔ͜ͱɺਓྗ࡞ۀΛݮΒ͢͜ͱ͕ࢦ਑ ΦϖϨʔγϣϯͷઃܭ
90. Control Loop Kubernetes ͷࠜװʹ Control Loop ͱ͍͏࢓૊Έ͕͋ΔɻKubernetes ͸ pod ͳͲͷ resource Λ؅ཧ͢Δࡍʹɺdesired state(ཧ૝ঢ়ଶ)ͱ actual state(࣮ࡍͷঢ়ଶ)ͷ 2 ͭΛ͓࣋ͬͯΓɺactual state Λ desired state ʹۙ ͚ͮΑ͏ͱ͢ΔॲཧΛӬٱʹ܁Γฦ͍ͯ͠Δɻ
91. Control Loop ཧ૝ͷঢ়ଶͱ࣮ࡍͷঢ়ଶͷൺֱ (Diff) ࣮ࡍͷঢ়ଶΛ֬ೝ (Observe) ཧ૝ঢ়ଶʹ͚ۙͮΔॲཧΛ࣮ߦ (Act)
92. Desired ίϯςφͷ਺: 5 Kubernetes ར༻ऀ͕ཧ૝ঢ়ଶΛ఻͑Δ ίϯςφͷ਺Λཧ૝ͷঢ়ଶʹ͚͍ۙͮͯ͘ྫ
93. Actual ίϯςφͷ਺: 2 Desired ίϯςφͷ਺: 5 ࣮ࡍͷঢ়ଶΛ֬ೝ (Observe) ίϯςφͷ਺Λཧ૝ͷঢ়ଶʹ͚͍ۙͮͯ͘ྫ
94. ίϯςφͷ਺Λཧ૝ͷঢ়ଶʹ͚͍ۙͮͯ͘ྫ Actual ίϯςφͷ਺: 2 Desired ίϯςφͷ਺: 5 ཧ૝ͷঢ়ଶͱ࣮ࡍͷঢ়ଶͷൺ ֱ(Diff)
95. Actual ίϯςφͷ਺: 5 Desired ίϯςφͷ਺: 5 ཧ૝ঢ়ଶʹ͚ۙͮΔॲཧΛ࣮ߦ (Act) ίϯςφͷ਺Λཧ૝ͷঢ়ଶʹ͚͍ۙͮͯ͘ྫ
96. એݴతͳΞϓϩʔνΛऔΔ ࣮ࡍͷঢ়ଶΛཧ૝ঢ়ଶʹ͍ͯ͘͜͠ͱɺͦͯͦ͠ͷํ๏(HOW) ʹ੹೚Λ࣋ͭͷ͕ Kubernetes ɻͦΕʹରͯ͠ཧ૝ঢ়ଶΛܾΊ(WHAT)ɺKubernetes ʹ఻͑Δͷ͕ Kubernetes ར༻ऀͷ੹೚ɻཧ૝ঢ়ଶΛ఻͑ؒҧ͑͹໰୊͕ى͖ͯ͠·͏ͨΊɺཧ ૝ঢ়ଶΛόʔδϣϯ؅ཧ͢Δɻ·ͨɺKubernetes ར༻ଆ͕ HOW ͷ෦෼ΛӅṭ(ந ৅Խ) ͯ͘͠Ε͍ͯΔͷ΋ؔΘΒ໋ͣྩతʹΦϖϨʔγϣϯ͢Δͷ΋ຊདྷͷઃܭҙ ਤʹ൓͢Δɻ
97. એݴతͳΞϓϩʔνΛऔΔ The declarative approach is key to the system’s self-healing and autonomic capabilities. Kubernetes Design and Architecture Ҿ༻
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE
98. એݴతͳΞϓϩʔνΛऔΔ In particular, it should be straightforward (but not required) to manage declarative intent under version control, which is standard industry best practice and what Google does internally. Version control facilitates reproducibility, reversibility, and an audit trail. ... Version control enables the use of familiar tools and processes for change control, review, and conflict resolution. Declarative application management in Kubernetes Ҿ༻
    IUUQTEPDTHPPHMFDPNEPDVNFOUED-1(XF7&:S7R2W#-+HTY75S&3N./0#"@DY;186
99. ͳͥ Kubernetes ͕ YAML ϕʔεͳͷ͔ 1. DSL ʹൺ΂ͯଟ͘ͷݴޠͰαϙʔτ͞Ε͍ͯΔ 2. Lint πʔϧ΋ଟ͘ଘࡏ͢Δ 3. API schema ͷৄࡉΛ֮͑ͳ͍ͱॻ͚ͳ͍΋ͷͷ… 1. ஗͔Εૣ͔Ε API schema ΍ Kubernetes ͷ֓೦ʹֶ͍ͭͯͼͨ͘ͳΔɺ· ͨ͸ֶͿඞཁ͕ग़ͯ͘ΔͷͰແବʹͳΒͳ͍ 2. Ή͠ΖҰ؏ੑ͕͋ͬͯΑ͍ ࢀߟ
    IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE
100. YAML ϚχϑΣετ͕ॻ͚ΔͳΒ… 
     FDIP
     b
     BQJ7FSTJPO
     BQQTWCFUB
     LJOE
     %FQMPZNFOU
     NFUBEBUB
     
     
     OBNF
     EFQMPZNFOUFYBNQMF
     TQFD
     
     
     SFQMJDBT
     
     
     
     SFWJTJPO)JTUPSZ-JNJU
     
     
     
     UFNQMBUF
     
     
     
     
     NFUBEBUB
     
     
     
     
     
     
     MBCFMT
     
     
     
     
     
     
     
     
     BQQ
     OHJOY
     
     
     
     
     TQFD
     
     
     
     
     
     
     DPOUBJOFST
     
     
     
     
     
     
     
     OBNF
     OHJOY
     
     
     
     
     
     
     
     
     JNBHF
     OHJOY
     
     
     
     
     
     
     
     
     QPSUT
     
     
     
     
     
     
     
     
     
     DPOUBJOFS1PSU
     
     
     c
     LVCFDUM
     DSFBUF
     G
     
101. REST API ΋࢖͑ΔΑ͏ʹͳ͍ͬͯΔ 
     DVSM
     9
     1045
     )
     $POUFOU5ZQF
     BQQMJDBUJPOZBNM
     EBUB
     
     BQJ7FSTJPO
     BQQTWCFUB
     LJOE
     %FQMPZNFOU
     NFUBEBUB
     
     
     OBNF
     EFQMPZNFOUFYBNQMF
     TQFD
     
     
     SFQMJDBT
     
     
     
     SFWJTJPO)JTUPSZ-JNJU
     
     
     
     UFNQMBUF
     
     
     
     
     NFUBEBUB
     
     
     
     
     
     
     MBCFMT
     
     
     
     
     
     
     
     
     BQQ
     OHJOY
     
     
     
     
     TQFD
     
     
     
     
     
     
     DPOUBJOFST
     
     
     
     
     
     
     
     OBNF
     OHJOY
     
     
     
     
     
     
     
     
     JNBHF
     OHJOY
     
     
     
     
     
     
     
     
     QPSUT
     
     
     
     
     
     
     
     
     
     DPOUBJOFS1PSU
     
     
     IUUQBQJTBQQTWOBNFTQBDFTEFGBVMUEFQMPZNFOU
102. YAML Λॻ͜͏ 1. DSL ΍ GUI ౳Ͱந৅Խ͞ΕͨϚχϑΣετΛॻ͘ͷ͸࣮͸ԕճΓͰ͸ͳ͍͔ 2. YAML Λॻ͘ͷ͸ΤϯδχΞϦϯάͰ͸ͳ͍ͱ͍͏ҙݟʹରͯ͠ 1. YAML Λॻ͘࡞ۀࣗମ͸ΤϯδχΞϦϯάͰ͸ͳ͍͔΋͠Εͳ͍͕ɺͦΕ͸ YAML Λॻ͘࡞ۀ෦෼͔͠ݟ͍ͯͳ͍ͱݴ͑Δ 2. ΤϯδχΞϦϯά͸ Kubernetes ʹΑͬͯ YAML ϑΝΠϧͷཪଆʹӅṭ͞Ε͍ͯΔ 3. YAML Λॻ͚ͩ͘ͰΠϯϑϥपΓͷ࡞ۀ͕׬݁͢Δɺखಈ࡞ۀ͕ෆཁʹͳ͍ͬͯΔ 4. ͦͷු͍ͨ࣌ؒͰผͷΤϯδχΞϦϯάλεΫΛ΍Δ΂͖Ͱ͋ΓɺYAML Λॻ͖ͨ ͘ͳ͍͔Β REST API Λ࢖ͬͯந৅Խ͢Δ౳ͷߦҝ͸ຊ຤స౗Ͱ͸ͳ͍͔
103. 1 Ϧιʔε 1 YAML ϑΝΠϧ 1. ͲͷϑΝΠϧʹͲͷϦιʔε͕ఆٛ͞Ε͍ͯΔ͔໌֬Ͱ͋Δ 2. ͲͷϦιʔεʹରͯ͠ΦϖϨʔγϣϯ͢Δͷ͔໌֬Ͱ͋Δ 3. ࠶ར༻ੑ͕ߴ͍ 4. ϦιʔεΛ௥Ճ͢Δ࣌ʹͲ͜ʹॻ͘΂͖͔໌֬Ͱ͋Δ ࢀߟ
     IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTBSDIJUFDUVSFBSDIJUFDUVSFNE
104. ϞχλϦϯάͷઃܭ
105. ͳͥϞχλϦϯά͢Δͷ͔ 1. γεςϜ͕ਖ਼ৗʹՔಇ͍ͯ͠Δ͔Λ೺Ѳ͢ΔͨΊ 2. Ͱ͸ɺͲͷγεςϜΛϞχλϦϯάͨ͠Βྑ͍͔ʁ
106. Kubernetes ͷੈքͰ͸γεςϜ͸ 2 छྨʹେผͰ͖Δ Πϯϑϥڞ௨ج൫ܥ ΞϓϦέʔγϣϯܥ
107. ϞχλϦϯά͢ΔγεςϜ͸େ͖͘ 2 ͭ Πϯϑϥڞ௨ج൫(Kubernetes) ΞϓϦέʔγϣϯ
108. ΞϓϦέʔγϣϯ͕ਖ਼ৗʹՔಇ͍ͯ͠Δͱ͸ʁ 1. ΞϓϦέʔγϣϯίϯςφ͕ىಈ͓ͯ͠Γɺ 2. ҰఆͷϨεϙϯελΠϜ಺Ͱɺ 3. ਖ਼ৗͳϨεϙϯε͕ฦ͍ͤͯΔঢ়ଶ
109. ΞϓϦέʔγϣϯͷϞχλϦϯά 1. ΞϓϦέʔγϣϯίϯςφ͕ਖ਼ৗʹىಈ͍ͯ͠Δ͔ 2. ҰఆͷϨεϙϯελΠϜ಺Ͱ͋Δ͔ 3. ਖ਼ৗͳϨεϙϯε͕ฦ͍ͤͯΔ͔
110. ڞ௨ج൫(Kubernetes)͕ਖ਼ৗʹՔಇ͍ͯ͠Δͱ͸ʁ 1. Kubernetes Master ͱ Nodes ͕ىಈ͓ͯ͠Γɺ 2. ༩͑ΒΕͨ desire state (ཧ૝ͷঢ়ଶ) ௨Γʹίϯςφ΍ॾʑͷϦ ιʔε͕ଘࡏ͍ͯ͠Δ͜ͱ
111. ڞ௨ج൫(Kubernetes)ͷϞχλϦϯά 1. Kubernetes Master ͱ Nodes ͕ਖ਼ৗʹىಈ͍ͯ͠Δ͔ 2. Desire state (ཧ૝ͷঢ়ଶ) ͱ Actual state (࣮ࡍͷঢ়ଶ) ʹဃ཭͕ͳ ͍͔
112. ͜͜·Ͱ͕ୈҰεςοϓ
113. Ͱ͸ਖ਼ৗՔಇ͍ͯ͠ͳ͍ͱ͖ʹͲ͏͢Δ͔ʁ 1. γεςϜ͕ਖ਼ৗՔಇ͍ͯ͠Δ͔Λ൑ผ͢Δσʔλ(ϝτϦΫε)Λݟ ͍ͯͯ΋ݪҼ͸Θ͔Βͳ͍ 2. γεςϜ͕ਖ਼ৗՔಇ͍ͯ͠ͳ͍ͱ͖͸ɺͦͷγεςϜͷߏ੒ཁૉͷ ͍ͣΕ͔Ͱ໰୊͕ੜ͍ͯ͡Δ 3. ͦͷߏ੒ཁૉ΋ϞχλϦϯά͓ͯ͘͜͠ͱͰݪҼڀ໌ʹܨ͛ΒΕΔ 4. ΞϓϦέʔγϣϯͱڞ௨ج൫(Kubernetes)ͷߏ੒ཁૉ͸ʁ
114. ΞϓϦέʔγϣϯΛߏ੒͢Δ΋ͷ ΞϓϦέʔγϣϯܥ CDN ϩʔυόϥϯα ΞϓϦέʔγϣϯ ϛυϧ΢ΣΞ Ϋϥ΢υαʔϏε ίϯςφ Namespace etc
115. Πϯϑϥڞ௨ج൫Λߏ੒͢Δ΋ͷ Πϯϑϥڞ௨ج൫ܥ Ϋϥ΢υαʔϏε Kubernetes Master Kubernetes Nodes kubelet kube-proxy kube-dns ϩάऩू daemon etc
116. 1. ϝτϦΫε 2. (τϨʔε) 3. Πϕϯτͱϩά ֤ߏ੒ཁૉʹؔͯ͠ 2 (3) छྨͷσʔλΛूΊΔ
117. ϝτϦΫε 1. WHAT(Կ͕ى͖͍ͯΔ͔)Λ೺Ѳ͢Δ΋ͷ 2. ग़དྷΔݶΓ৭ʑͳϝτϦΫεΛूΊ͓ͯ͘ 3. ޙ͔ΒλάɺϥϕϧͰඞཁͳ΋ͷΛݟΔ 4. ྫ: Desired Pod ਺ɺCPU ࢖༻཰ɺϨεϙϯελΠϜ
118. τϨʔε 1. HOW (Ͳ͏ͳ͍ͬͯΔ͔)Λ೺Ѳ͢Δ΋ͷ 2. ϝτϦΫεΑΓ΋ৄࡉʹͲ͜Ͱ໰୊͕ى͖͍ͯΔ͔Θ͔Δ 3. ྫ: ͲͷΫΤϦ͕஗͍ͷ͔ɺͲͷؔ਺͕Τϥʔʹͳ͍ͬͯΔ ͷ͔
119. Πϕϯτͱϩά 1. WHY (ͳͥى͖͔ͨ)Λ೺Ѳ͢Δ΋ͷ 2. ϝτϦΫε΍τϨʔε͔Β͸Θ͔Βͳ͍໰୊ͷݪҼ΍ى఺͕ Θ͔Δ 3. ྫ: ΦʔτεέʔϧΠϕϯτɺΞΫηεϩά
120. ϞχλϦϯά·ͱΊ 1. ίϯςφͷϥΠϑαΠΫϧͷ୹͞ɺಈతͳ഑ஔɺࣗ཯తͳڍಈͱɺ Observability(ࢹೝੑ) ͕ͳ͍ͱԿ͕ى͍ͬͯ͜Δ͔Θ͔Βͳ͍ 2. ೝࣝͰ͖ͳ͍΋ͷ͸վળͰ͖ͳ͍ͨΊɺܧଓతͳ Kubernetes ӡ ༻ͷվળʹϞχλϦϯά͸ඞཁෆՄܽͰ͋Δ
121. ·ͱΊͱิ଍
122. 1. Kubernetes Λ࢖ͬͯԿΛ࣮ݱ͍ͨ͠ͷ͔ɺKubernetes ͷઃܭऀ͸ԿΛҙਤͯ͠࡞ͬͨ ͷ͔Λҙࣝ͢Δͱɺݸʑͷػೳͷ࢖͍ํ΍ཁૉͷଘࡏҙ͕ٛݟ͑ͯ͘Δ 2. ΫϥελͷωοτϫʔΫઃܭ΍ϊʔυपΓɺηΩϡϦςΟɺϞχλϦϯάৄࡉ౳৮ΕΒ Εͯͳ͍߲໨΋͋Δ͕ɺεϥΠυຕ਺͕๲େʹͳ͖ͬͯͨͷͰࠓճ͸͜͜·Ͱ… 3. ͜͜Ͱ঺հͨ͠ઃܭ΍ϓϥΫςΟεΛશͯద༻͠Α͏ͱࢥ͏ͱ೔͕฻ΕΔɻ͍ͭ·Ͱܦͬ ͯ΋ຊ൪ӡ༻͕։࢝Ͱ͖ͳ͍ͷͰɺࣄલʹ΍Δ͜ͱͱɺࣄޙʹ΍Δ͜ͱͷόϥϯεΛऔ Γ·͠ΐ͏ 1. Ͳ͕͜ڥք͔ͱ͍͏ͱɺຊ൪ӡ༻։࢝ޙʹɺࣗಈԽ΍͜͜ʹॻ͍ͨϓϥΫςΟεΛ ద༻Ͱ͖Δ༨༟(શମͷ50%)͕ग़དྷΔ͘Β͍ͷ४උΛ͢Δͱ͍͍ͱࢥ͏ 2. શ͘४උͤͣʹຊ൪ӡ༻Λ։࢝͢Δͱ໰୊ͷରԠ΍ख࡞ۀʹຒ΋ΕͯෛͷεύΠϥ ϧʹؕΔ ·ͱΊͱิ଍
123. End