スケール可能で運用しやすいWebSocketサーバ設計の課題と工夫 / The development of scalable and operating friendly WebSocket Server

CEDEC 2018

Transcript

1. εέʔϧՄೳͰ ӡ༻͠΍͍͢ WebSocketαʔόઃܭͷ ՝୊ͱ޻෉ CEDEC 2018 ໘ന๏ਓΧϠοΫɹ୩࿬ ਅۏ
2. ͜ͷηογϣϯͰ࿩͢͜ͱ • ैདྷͷWebΞϓϦέʔγϣϯͷ࡞ΓํͷԆ௕ઢ্Ͱ • ϛυϧ΢ΣΞΛ෇͚Ճ͑ΔܗͰWebSocketʹΑΔϦΞϧ λΠϜ௨৴Λ༻͍Δ • ήʔϜαʔόΛ։ൃɾӡ༻ͯ͠Έͨ ͦͷ࣌ͷ޻෉ͱۤ࿑Λ࿩͠·͢ɻ
3. ͜ͷηογϣϯͰ࿩͞ΕΔ͜ͱ͸ ౦ژϓϦζϯ1ͱ͍͏ϞόΠϧήʔϜͷࣄྫͰ͢ 1 https://tokyo-prison.kayac.com/
4. ήʔϜͷಛੑ • ൒λʔϯ੍ɻλʔϯ͸Ұఆඵ਺͕ܦա͢Δͱऴྃ͢Δɻλʔϯ ಺͸͋Δఔ౓ࣗ༝ʹಈ͚Δ
5. ήʔϜͷಛੑ • ϓϨΠϠʔ3ਓ vs ϓϨΠϠʔ3ਓ
6. ߨԋऀͷόοΫάϥ΢ϯυ • ໘ന๏ਓΧϠοΫ ιʔγϟϧήʔϜࣄۀ෦ • αʔόαΠυΛઐ໳Ͱ୲౰ɻΞϓϦέʔγϣϯίʔυΛॻ͕͘ Πϯϑϥߏங΋΍Γɺӡ༻΋΍Δ • झຯ͸Perlͱ3DϓϦϯλվ଄ͱϛυϧ΢ΣΞ։ൃ
7. ैདྷͷWebΞϓϦͰ ΞʔΩςΫνϟͰ WebSocketΛ༻͍Δͱ͖ͷ՝୊
8. આ໌ͷॱ൪ 1. ैདྷͷWebΞϓϦͷߏ଄ 2. WebSocketΛ࢝Ίͱͨ͠ੜଘ࣌ؒͷ௕͍ηογϣϯΛѻ͏্Ͱ ͷ໰୊ 3. ղܾํ๏(kuiperbeltͷ঺հ)΁
9. ͜͜Ͱ͍͏WebΞϓϦͱ͸ • ΠϯλʔωοτΛ௨ͬͯओʹHTTPΛ࢖͏ ΞϓϦέʔγϣϯ • HTTP͸·ͣϦΫΤετ͋Γ͖
10. ͳͥHTTPΛ࢖͏ͷ͔ • ͍͍ͩͨͷWebϑϨʔϜϫʔΫ͸HTTPΛѻ͏ͨΊʹ࡞ΒΕ͍ͯ Δ • curlΛ͸͡Ίͱͨ͠HTTPΛσόοά͢ΔͨΊͷπʔϧ΍஌ݟ͕ ੈͷதʹ͋;Ε͍ͯΔ • HTTP͸εςʔτϨε͕લఏͷϓϩτίϧͳͷͰεέʔϧͤ͞Δ ͷʹ༗ར(ޙड़)
11. εςʔτϨε͔ͩΒԣʹ૿΍ͤΔ Keep-AliveͰTCP઀ଓ࢖͍ճ͠Λߦ͏৔߹΋͋Δ͕ɺHTTPͷ૚͔ΒݟΕ͹ಉ͡
12. εςʔτϨε͔ͩΒԣʹ૿΍ͤΔ
13. Ϣʔβ਺෼ಉ࣌ʹϦΫΤετΛड͚͍ͯΔ͜ ͱ΋ͳ͍
14. WebΞϓϦέʔγϣϯͷੈք • HTTP͕εςʔτϨεͳͷͰԣʹಉ͡ΞϓϦΛฒ΂ͯLBʹೖΕ Δ • ෛՙ෼ࢄͱHA໨త • Lightweight Language(PHP΍Ruby2ͳͲ)͕࢖ΘΕΔ͜ͱ͕͋Δ • ͜ΕΒ͸ϚϧνϓϩηεϞσϧ͕ओྲྀ 2 ΋ͪΖΜJava΍C#Ͱ૊Ήͱ͜Ζ΋͋ΓɺͦΕΒ͸ϚϧνεϨου
15. ϦΞϧλΠϜ௨৴ΛWebͷԆ௕Ͱ΍Δબ୒ࢶ • Long Polling • WebSocket • WebRTC DataChannel • ͋Δ͍͸ੜιέοτ
16. ͳͥWebSocketΛ࢖͏ͷ͔ => ௕͍΋ͷʹר͔Εͨ΄͏ָ͕͕Ͱ͖Δ • 2011೥ʹඪ४ࡦఆ͞Ε͔ͯΒ͕࣌ؒܦͪϥΠϒϥϦ͕ރΕ͍ͯ Δ • ੜιέοτͱҧͬͯϥΠϒϥϦΛࣗ෼Ͱॻ͔ͳͯ͘Α͍ • wscatΛ࢝Ίͱͨ͠σόοάπʔϧ΋๛෋ • TCPͳͷͰ࠶ૹͳͲͷ໘౗͸ݟͯ͘ΕΔ
17. ͦ΋ͦ΋ͳͥඪ४Webٕज़ʹͩ͜ΘΔͷ͔ • Web։ൃऀίϛϡχςΟʹ৐͔ͬΕΔ • ϥΠϒϥϦ΍஌ݟΛٵऩͰ͖Δ • (͜Ε͸ޙ෇Ͱ͕͢)PWAήʔϜΞϓϦ࣌୅Λݟਾ͑ͯͷબ୒ • ϒϥ΢β͸࢖͑Δϓϩτίϧ͕ݶΒΕΔ
18. WebSocket͸εςʔτϑϧ௨৴ • ΫϥΠΞϯτ͸WebAppʹରͯ͠ܨ͗ͬͺͳ͠ͷঢ়ଶʹͳΔ • WebSocket͸ηογϣϯͷੜଘ͕࣌ؒ௕͍ͨΊͦͷؒ͸Ϧ ιʔεΛઐ༗͢Δ
19. WebSocket͸઀ଓΛҡ࣋͠ଓ͚Δඞཁ͕͋Δ
20. ϦΞϧλΠϜ௨৴͸εςʔτϑϧ௨৴ • 1ͭͷΫϥΠΞϯτ͕1WebAppΠϯελϯεͱີ݁߹͢Δ • গͳ͘ͱ΋1ηογϣϯͷؒ͸ॊೈͳεέʔϧΞ΢τ(୆਺Λ ԣʹ૿΍͢)΍ϦΞϧλΠϜ௨৴͕ߦΘΕ͍ͯΔؒͷίʔυ ߋ৽(σϓϩΠ)͕೉͍͠
21. ϦΞϧλΠϜ௨৴͸εςʔτϑϧ௨৴ • εςʔτϑϧ௨৴ͱಉظܕPreforkͷ૬ੑͷѱ͞ • 1ݸͷϓϩηε͕1ͭͷϦΫΤετΛಉظతʹॲཧ • 1ϦΫΤετͷॲཧ͕࣌ؒ୹͍(ʙ1sec)͜ͱ͕લఏ • 1ϓϩηεͰ1WebSocketηογϣϯΛѲΔͱϓϩηε਺෼͠ ͔ΫϥΠΞϯτΛऩ༰Ͱ͖ͳ͍
22. ٧·Δਤ
23. ࠓ·ͰͷWebΞϓϦ͕ͦͷ·· WebSocketΛ஻Δ͜ͱ͸೉͍͜͠ͱ ͕Θ͔ͬͨ
24. ͍͔ʹͯ͠զʑ͸ϦΞϧλΠϜ௨৴ʹཱͪ޲ ͔͑͹͍͍ͷ͔ • ָͰ • Θ͔Γ΍ͯ͘͢ɺγϯϓϧͰ • εέʔϧՄೳͰ • ӡ༻͠΍͍͢ͷ͕΄͍͠
25. ͦΜͳ͏·͍ ιϦϡʔγϣϯ͸͋Δͷ͔
26. ैདྷͷWebΞʔΩςΫνϟ͕WebSocketΛ ѻ͏ͱ͖ͷ೉͍͠఺Λ੔ཧ • ௕࣌ؒੜ͖ଓ͚Δ௨৴ηογϣϯΛҡ࣋͢Δͷ͕೉͍͠ • ௨৴ͱϢʔβΛඥ෇͚Δͷ͕೉͍͠ => ͜ͷลϛυϧ΢ΣΞʹ೚ͤΔͷ͸Ͳ͏͔ ֎ʹग़͢ͷ͸ϚΠΫϩαʔϏεͷߟ͑ํʹ͍͔ۙ΋͠Εͳ͍
27. WebSocketηογϣϯΛ ҡ࣋͢Δ͚ͩͷ ϛυϧ΢ΣΞ kuiperbeltͷ։ൃͱಋೖ
28. kuiperbeltͱ͸ • OSSͰ͢3 • WebSocketͷηογϣϯΛड͚ͯҡ࣋͢Δ͚ͩͷ௨৴σʔϞϯ • ࠷ॳ͸ݸਓϓϩδΣΫτͰ͕ͨ͠ɺձࣾͰ͔ͳΓ࢖ΘΕ͍ͯ· ͢ • ձࣾͷਓʹ΋ίϛοτͯ͠΋Β͍ͬͯ·͢ 3 https://github.com/kuiperbelt/kuiperbelt
31. ໊લͷ༝དྷ ఱจֶͷΤοδϫʔεɾΧΠύʔϕϧτ*͔Β * photo by NASA
32. kuiperbelt͕ग़དྷΔ͜ͱ • WebSocketηογϣϯͷҡ࣋ • HTTP-APIͰͷಛఆηογϣϯ΁ͷϝοηʔδͷྲྀ͠ࠐΈ • ্Γ(ΫϥΠΞϯτ->αʔό)͸ඇରԠ7 7 ରԠ༧ఆ͸͋Γ
33. ࿩͢͜ͱͷྲྀΕ 1. ౦ژϓϦζϯͰͷαʔόߏ੒ͷུ֓ 2. kuiperbeltΛ࢖ͬͨWebSocketηογϣϯͷೝূɾϝοηʔδ഑ ৴ 3. ઃܭࢥ૝ 4. ࣮ࡍͷӡ༻ࣄྫ 5. ౦ژϓϦζϯͰͷ࣮ࡍͷ࢖ΘΕํ
34. ౦ژϓϦζϯͷαʔόߏ੒8 8 ຊ೔17:50ΑΓฐࣾ౻ݪͷৄ͍͠ߏ੒ͷղઆ͕͋Γ·͢
35. HTTPͷதͷ࢓૊ΈͷதͰೝূΛ͢Δ • ೝূ͸WebSocket΁ͷΞοϓάϨʔυલʹWebAppଆʹϓϩΩ γΛߦ͏ • ϓϩΩγ࣌ʹkuiperbelt͸ࣗ෼ͷղܾՄೳͳϗετ໊Λ෇Ճ͢ Δ -> εέʔϧΞ΢τʹ༻͍Δ • WebAppଆ͸ϨεϙϯεͷϔομʹࣝผࢠΛՃ͑Δ
36. HTTPͷதͷ࢓૊ΈͷதͰೝূΛ͢Δ
37. HTTPͷதͷ࢓૊ΈͷதͰϝοηʔδ഑৴Λ ͢Δ • ΫϥΠΞϯτ΁ͷϝοηʔδ͸ϔομͰࣝผࢠΛࢦఆͯ͠ɺϘ σΟͷ಺༰͸ͦͬ͘Γͦͷ··WebSocketͷηογϣϯʹྲྀ͠ ࠐΉ • த਎ͷϝοηʔδͷܗࣜ͸kuiperbelt͸ؔ༩͠ͳ͍ • ౦ژϓϦζϯ͸όΠφϦΛྲྀͯ͠·͢
38. HTTPͷதͷ࢓૊ΈͷதͰϝοηʔδ഑৴Λ ͢Δ
39. kuiperbeltࣗ਎ͷઃܭࢥ૝ • ୯७ͳ͜ͱ͔͠΍Βͳ͍ͷͰམͪͳ͍ɾ࠶ىಈ͠ͳͯ͘ྑ͍ • ήʔϜϩδοΫ͸஌Βͳ͍ͷͰσϓϩΠ͕ඞཁͳ͍ • ൒೥Ҏ্ӡ༻͍ͯ͠Δ͕ϝϞϦϦʔΫ͸ى͖͍ͯͳ͍ • ಛఆϓϩτίϧɾϛυϧ΢ΣΞʹґଘ͠ͳ͍
40. ผλΠτϧͰͷো֐ࣄྫ ͏͔ͬΓHTTP/2ࣄ݅ ݱࡏ͸ҰఆִؒͰίωΫγϣϯΛഁغ͢Δ࣮૷͕ೖ͍ͬͯ·͢
41. kuiperbeltࣗ਎ͷઃܭࢥ૝ • GoͰॻ͔Ε͍ͯΔͷͰίΞ෼͚ͩͪΌΜͱεέʔϧ͢Δ • 1ίΞ͋ͨΓඵؒ10,000ϝοηʔδ͙Β͍(1ϝοηʔδ͋ͨΓ ฏۉ150bytes) • ϝϞϦྔ͸ΫϥΠΞϯτ਺ʹൺྫ͠ɺϝοηʔδͷ಺༰ͳͲ ʹ΋ࠨӈ͞ΕΔ͕1ສ઀ଓͰ1GBʙ1.8GB
42. ࣮ࡍͷӡ༻Ͱ͸ • ࣮ෛՙࢼݧͰϝοηʔδྲྀྔͱϝοηʔδαΠζ͕େ͖͍͜ͱ ͕൑໌ͨͨ͠Ίɺ1Πϯελϯε5000઀ଓఔ౓ʹ཈͑ͨ • m3.large4ͱr4.large5ͷspot fleetʹ͍ͯͨ͠Βm3.largeͷpps ্ݶ͕௿ͯ͘ภΓ͕ग़ͨͷͰr4.largeͷΈͷӡ༻ʹ͍ͯ͠Δ 5 2core 15.25GBϝϞϦ 4 2core 7.5GBϝϞϦ
43. ύέοτ٧·ΓΛىͯ͜͠࠶ૹ͕ߦΘΕ͍ͯ Δϩά 21:00:47 plack.1 | 1019.259 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 21:00:47 plack.1 | 3017.341 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 21:00:47 plack.1 | 1017.801 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 21:00:47 plack.1 | 7174.359 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 21:00:47 plack.1 | 1025.966 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 21:00:47 plack.1 | 10010.330 ms [Furl::HTTP] POST http://XX.XX.XX.XX:9180/send | Furl:40 1, 3, 7, 10...ͱ஗Ԇ͍ͯ͠ΔͷͰ࠶ૹͰ٧·͍ͬͯΔͷͰ͸ʁ ͱ ͳͬͨ
44. ΍͍ͬͯΔ͜ͱ͕γϯϓϧͳͷͰ༷ʑͳΞʔΩςΫνϟʹରԠͰ͖Δ MO޲͖ͷதܧ͚ͩߦ͏ΞʔΩςΫνϟ
45. ΍͍ͬͯΔ͜ͱ͕γϯϓϧͳͷͰ༷ʑͳΞʔΩςΫνϟʹରԠͰ͖Δ MMO΍Ϛονϯά޲͖ͷαʔόͰήʔϜϩδοΫΛ࣮ߦ͢ΔΞʔΩςΫνϟ
46. ϦΞϧλΠϜ௨৴αʔόΛϩʔυόϥϯαʔʹ૊ΈࠐΉ Q: ෳ਺ͷαʔόΛܦ༝͢Δ͔ΒϨΠςϯγ͕ݫ͍͠ͷͰ͸ʁ A1: ϞόΠϧʹ͓͍ͯ͸ͦͷݶΓͰ͸ͳ͍ A2: λʔϯ੍ͳΒ͋Δఔ౓ڐ༰Ͱ͖Δ
47. A1: ϞόΠϧʹ͓͍ͯ͸ͦͷݶΓͰ͸ͳ͍ HTTP-APIΛଧ͔ͬͯΒWebSocketͰԠ౴͕͔͑ͬͯ͘Δ·Ͱͷ RTTΛܭଌ6 6 JavaScriptͳͷͰspectreͷӨڹͰ2msఔ౓ͷؙΊ͕ଘࡏ͍ͯ͠Δͷ͸ཹҙ
48. A1: ϞόΠϧʹ͓͍ͯ͸ͦͷݶΓͰ͸ͳ͍ • લ൒͕ݻఆޫճઢܦ༝ͷWiFi, ޙ൒͕4G • WiFiͩͱ20 ʙ 30ms, 4Gͩͱ75msʙ • LBΛܦ༝͢Δ͜ͱʹΑΔϨΠςϯγ͸1ܻmsఔ౓ • ϞόΠϧʹ͓͍ͯ͸LBʹΑΔϨΠςϯγ͸ࢧ഑తͰ͸ͳ͍
49. A2: λʔϯ੍ͳΒ͋Δఔ౓ڐ༰Ͱ͖Δ • ͜Ε͕PC΍ίϯγϡʔϚͰ΍ΔΑ͏ͳFPS΍֨ήʔͰ͋Ε͹1ܻ msͷ஗ԆͰ͢Β୹ॖ͍ͨ͠ • ڐ༰͞ΕΔ஗Ԇ෯͸ήʔϜγεςϜʹΑͬͯมΘΔ • ཁ݅ʹΑͬͯ͸ૹΓͬͺͳ͠ʹͰ͖Δ/Φʔόʔϔου͕গͳ ͍UDPͳͲΛબ୒͢Δ
50. ݱঢ়͸LBʹೖΕΔϝϦοτ͕Ͱ͔͍ • TLS TerminationΛ೚ͤΔ͜ͱ͕ग़དྷΔ • ݸผʹTerminationΛ΍͍ͬͯΔͱূ໌ॻͷ؅ཧ͕໘౗ • ࠷ۙ͸௨Βͳ͍ϑΥϫʔυϓϩΩγ͕ݮͬͨͱ͸͍͑ੜ WebSocket͸ૄ௨͢Δ͔Ͳ͏͔ͷϦεΫ͕͋Δ
51. ݱঢ়͸LBʹೖΕΔϝϦοτ͕Ͱ͔͍ • ࣗಈͰ༙͍ͨԾ૝Ϛγϯ/ίϯςφʹ౸ୡՄೳͳIPΞυϨε/ϗ ετΛͲ͏΍ͬͯΫϥΠΞϯτʹ௨஌͢Δ͔໰୊ • ैདྷ͸ಉ͡ϧʔϜΛݻΊΔΑ͏ʹϚονϯά࣌ʹ઀ଓ͢΂͖ ΞυϨεΛࣔͨ͠Γ • LBͳΒೖΕΔ͚ͩͰ౸ୡՄೳʹͳΔɺϧʔϜͰݻΊΔΈ͍ͨ ͳͷ͸ग़དྷͳ͍͕
52. ࣮ࡍʹ૊ΈࠐΜͰΈͯ౦ژϓϦζϯ ͸ͲͷΑ͏ʹήʔϜΛਐߦ͍ͯ͠Δ ͔
53. kuiperbeltΛ૊ΈࠐΜͩ౦ژϓϦζϯͷॲཧ
54. kuiperbeltΛ૊ΈࠐΜͩ౦ژϓϦζϯͷॲཧ
55. ήʔϜϩδοΫ͕αʔόଆʹ͋Δ͜ͱͷϝ Ϧοτ • ήʔϜϩδοΫͷมߋ͕͋Δఔ౓༰қ ετΞਃ੥͕ཁΒͳ͍ • ΫϥΠΞϯτੑೳʹҾͬுΒΕͳ͍ • ΫϥΠΞϯτ։ൃऀ͸ֆΛग़͢͜ͱʹूதͰ͖Δ • νʔτ΋͞Εʹ͍͘ • ϩΪϯά΋༰қͳͷͰղੳͳͲ΋Ͱ͖Δ
56. ήʔϜϩδοΫ͕αʔόଆʹ͋Δ͜ͱͷσϝ Ϧοτ • ΫϥΠΞϯτ͕αʔόͷॲཧΛ଴ͭͨΊϨεϙϯγϏϦςΟ͕ ѱ͍ • αʔόͷϨεϙϯεΛ଴ͨͳ͍ͳͲͷରࡦΛߨ͍ͯ͡Δ(ඇಉ ظԽ)
57. ήʔϜϩδοΫ͕αʔόଆʹ͋Δ͜ͱͷσϝ Ϧοτ • αʔόͷෛՙରࡦΛे෼ʹߦ͏ඞཁ͕͋ΔɹϦιʔε΋֬อ͢ Δඞཁ͕͋Δ • ૝ఆ࣮ෛՙࢼݧͱνϡʔχϯάΛ܁Γฦ͠·͕ͨ͠ඇৗʹۤ ࿑͠·ͨ͠
58. ͦͷଞkuiperbeltΛಋೖͨ͜͠ͱͰݟ͖͑ͯͨ ໰୊఺ 1. ϝοηʔδ౸ୡॱ໰୊ 2. αʔόؒϝοηʔδܽଛ໰୊
59. ϝοηʔδ౸ୡॱ໰୊ kuiperbeltͷதͰϩοΫऔΕͨॱ൪ʹྲྀΕΔ
60. ϝοηʔδ౸ୡॱ໰୊ ղ๏ • γʔέϯε൪߸Λ΋্ͨͤͨͰ 1. ΫϥΠΞϯτଆͰόοϑΝϦϯάͭͭ͠ฒ΂௚͢ 2. kuiperbeltଆͰόοϑΝϦϯάͭͭ͠ฒ΂௚͢
61. ϝοηʔδ౸ୡॱ໰୊ ΫϥΠΞϯτଆͰฒ΂௚͢ͷΛ࠾༻ • kuiperbeltͰ΍ΔͱϝϞϦ࢖༻ྔ͕૿͑Δ, ঢ়ଶ͕૿͑ͯίϯς φ͕མͪͨͱ͖ʹϝοηʔδܽଛ͕૿͑Δ • ඈ͹ͦ͏ͱͨ͠ϝοηʔδ͸ͬ͞͞ͱૹͬͯঢ়ଶΛͳ͘͢ • kuiperbelt͸த਎ʹྲྀΕΔσʔλʹؔ༩͠ͳ͍ϧʔϧ
62. αʔόؒϝοηʔδܽଛ໰୊ • HTTPͰ௚઀ૹΔͱܽଛ͢ΔΜ͡Όͳ͍ʁʁʁ • DC಺௨৴, Ϧʔδϣϯ಺௨৴ͳͷͰܦ࿏ͰམͪΔ͜ͱ͸ͳ͔ ͳ͔ͳ͍ • ϦτϥΠ&ΫϥΠΞϯτଆͰͷॏෳݕ஌ͳͲ΋ೖΕ͍ͯΔ • ࠓޙ͸circuit breaker΋ೖΕΔ
63. ͦͯ͠ӡ༻͍ͯ͠Δࠓ͸ • αʔόͷਓ͕͋Μ·ΓWebSocketͷ͜ͱΛߟ͑ͣʹࠓ·ͰͲ͓ ΓίʔυΛॻ͚Δ • WebSocketαʔό͕ίϯςφͱϩʔυόϥϯαʔʹΑͬͯϊʔ ϝϯςͰಈ͍͍ͯΔ • ͍ͭͰ΋ίϚϯυҰൃͰεέʔϧΞ΢τՄೳ • WebSocketͷ͜ͱΛ͋·Γߟ͑ͣʹσϓϩΠ͕ग़དྷΔ
64. εέʔϧՄೳͰ ӡ༻͠΍͍͢ WebSocketαʔό ͕ಈ͍͍ͯΔ
65. ·ͱΊ • ैདྷͷWebΞϓϦέʔγϣϯΞʔΩςΫνϟʹൺֱత༰қʹಋ ೖͰ͖ΔϦΞϧλΠϜ௨৴༻ϛυϧ΢ΣΞΛ࡞࣮ͬͯࡍʹຊ൪ ಋೖ͠·ͨ͠ • ଞͷطʹӡ༻தͷήʔϜʹ΋ಋೖ͢Δχʔζ͕͋ΔΑ͏ͳͷͰ ͜Ε͔Β΋΍͍͖ͬͯ·͢ • ຊ൪ӡ༻ͯ͠ΈͯɺղܾՄೳͰ͸͋Δ͕ɺͦͦ͜͜ͷ໰୊͸͍ ͔ͭ͘߹ͬͨͷͰڞ༗͍͖ͤͯͨͩ͞·ͨ͠
66. Any Questions?