Share
Embed
ぜんぶ AWS でやらないワケ
DeNA では一部のサービスを AWS で運用している一方、大部分のサービスはオンプレミス DC で運用しています。このセッションでは、DeNA のサービスを AWS へ完全移行をすると仮定した場合に、直面しそうな困難や足りないと思われる機能などの、”ぜんぶ AWS でやらないワケ” をお話したいと思います。
Takumi Sakamoto
March 15, 2014
Transcript
61 slides
-
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ͥΜͿ"84ͰΒͳ͍ϫέ March
15, 2014 ! Takumi Sakamoto / @takus IT Platform Dept. System Infrastructure Management Gr. DeNA Co., Ltd. -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "CPVU.F ɾࡔຊາ5BLVNJ4BLBNPUP
ɾ!UBLVT ɾ0QFSBUJPO&OHJOFFS ɾ%F/"$P -5% d 2 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ࠷ۙ৭ʑͬͯ·͢ 5
ιʔγϟϧήʔϜࣄۀ ΤϯλʔςΠϝϯτࣄۀ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϝΠϯΦϯϓϨϛεڥ ɾ%F/"ͷαʔϏεͷଟ͘ΦϯϓϨͰӡ༻
ɾ౦ຊɾຊͷڌͰ%3ߏ "DU"DU ɾαʔόͷઍن 7 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ίετӡ༻Ͱ༏ҐͳΒ"84 ɾւ֎ࢢ͚ͷαʔόΛݱʹ༻ҙ͢Δ
ɾϐʔΫ࣌ͷ༻͕ಛʹߴ͍Ұ෦ͷαʔό ɾ৽نαʔϏεͰधཁ͕ಡΈʹ͍͘߹ 8 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ຊ͓͢Δ༰ ɾΦϯϓϨ"84྆ํͷϢʔβͱͯ͠
w γεςϜΞʔΩςΫνϟɾΦϖϨʔγϣϯͷհ ɾ͜ΕͰ࠷ڧͷ"84ʹ w "84͏্ͰͬͯΔ͜ͱͳͲ w "84Ͱ࣮ݱͰ͖ͨΒ͍͍͜ͱ 9 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. γεςϜͷجຊߏ ɾΦϯϓϨͰ"84Ͱجຊతͳߏಉ͡
ɾ8FC%BUBCBTF$BDIFͷߏ Ћ ɾECDBDIF%/4ϥϯυϩϏϯͰෛՙࢄ 11 Load Balancer web/app server web/app server Web/App mydns mysql memcached mysql MySQL memcached Memcached MyDNS Load Balancer -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϩʔυόϥϯα ɾΦϯϓϨͰ)BSEXBSF-#Λར༻
ɾ&-#ͱಉ༷ʹ"1*Ͱϝϯόͷૢ࡞͕Մೳ ɾߴՁ͕ͩେྔͷτϥϑΟοΫΛ҆ఆͯ͠ࡹ͚͍ͯΔ ɾγΣϧεΫϦϓτͰϔϧενΣοΫͱ͔ 12 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϩʔυόϥϯα ɾ"84ͰͪΖΜ&-#Λར༻
ɾ҆Ձʹར༻Ͱ͖ͯεέʔϧ͢Δ͕ݒ೦͋Δ ɾಥൃతͳΞΫηε૿ՃϦΞϧλΠϜ௨৴ ޙड़ 13 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. 8FC"QQ ɾΦϯϓϨ"84Ͱಉߏ
w 8FC"QBDIF/HJOY w "QQ1MBDL'BTU$(* 1FSM ɾ"QQͷ8"'Ͱ։ൃ͢Δ͜ͱ͕ଟ͍ ˞ 14 ※ DeNA͕։ൃͨ͠৽ͨͳϑϨʔϜϫʔΫGunyaSiFͱʁ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. .Z42- ɾΦϯϓϨ"84Ͱಉߏ
w .Z42- *OOP%# w .)"GPS.Z42-ʹΑΔϚελͷԽ w 2.)BOEMFS4PDLFUར༻ ɾ3%4ͱͷൺֱޙͰ 15 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. .FNDBDIFE ɾΦϯϓϨ"84Ͱಉߏ
w ಛघͳ͜ͱ͋·Γͯ͠ͳ͍ w &MBTUJD$BDIF͍ͬͯͳ͍ • ϩά͕औΕͳ͍ͳͲͷσϝϦοτ͕ଟ͍ w 3FEJT3FEJT4FOUJOFMͰԽͰ͖Δ 16 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. $%/ ɾ"LBNBJ$%/$MPVE'SPOU
w "LBNBJ͕ଟ͍͕$MPVE'SPOU͏͜ͱ͋Δ w Ձ֨ͱΤοδϩέʔγϣϯͷͳͲʹҧ͍͋Γ 17 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. &YUFSOBM%/4 ɾ"LBNBJ(5.3PVUF
w ΦϯϓϨͰ"LBNBJɺ"84Ͱ3PVUFΛར༻ ɾ"1*ͷ༷͕େ͖͘ҧ͏ͷ͕ಛత w "LBNBJκʔϯ୯ҐͰશ্ॻ͖͔͠Ͱ͖ͳ͍ w 3PVUFϨίʔυ୯ҐͰૢ࡞Ͱ͖Δ 18 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. جຊతͳํ ɾΦϯϓϨͱ"84Ͱ୯Ұͷӡ༻Λ࣮ݱ͍ͯ͠Δ
w ਓతϦιʔεͷྲྀಈੑ͕ߴ·Δ w ࢹεΫϦϓτͳͲྲྀ༻Մೳ w ϊϋͷڞ༗͕༰қ 20 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. (MPCBM*OGSB'SBNFXPSL ɾαʔόɾΦϖϨʔγϣϯͷͨΊͷπʔϧηοτ
w αʔόཧπʔϧ "ENJO5PPM w ֤छίϚϯυ αʔόϦετऔಘɺDSPOϥούʔ w ࢹπʔϧ༻εΫϦϓτ ઃఆϑΝΠϧࣗಈੜ w ศརϞδϡʔϧ MPHHFSɺΞϥʔτϝʔϧૹ৴ 21 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ΦϯϓϨɾ"84Ͱಉ͡ૢ࡞͕Մೳ #
=> get active server list get-hosts -u1 web -u2 ff ww0001 ww0002 ... ww2046 ! # => check diff diff_infra /etc/passwd /etc/passwd LOCAL a030159cf068b727b2a14398dcec6f63 wd0005 a030159cf068b727b2a14398dcec6f63 wd0007 641b11cf9a916ebf4ece86e55a3376c1 ! # => generate monitoring configurations make-nagios-conf -t $(get-hosts -u web:ff --delimtier ,) ! # => deploy global-infra resources dist-infra -x 22 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϓϩϏδϣχϯά ɾίϐʔݩͷαʔόΛෳͯ͠αʔόΛ࡞
w ίϐʔݩαʔόඞཁͳϛυϧΣΞ͕શೖΓ ɾ5PVSZP ߏཧπʔϧ ͰαʔόຖͷઃఆΛมߋ w ͋·Γෳࡶͳ͜ͱ͍ͯ͠ͳ͍ w ઃఆϑΝΠϧฤूɺσʔϞϯͷىಈͳͲ 23 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϓϩϏδϣχϯάͷྫ #
=> create instance from golden AMI ec2-create-instances -p ww -t c3.xlarge -z us-west-1b ! # => provisioning by touryo(edit config files, start daemons) touryo run ww2000 ww2000: Some run scripts executed. Test again... … ok 1 - httpd.conf ok 2 - daemontools apache 1..2 ok 10 - apache … All tests successful. ! # => deploy application dist-project -x ! # => register to load balancer update-elb -u1 web -u2 ff 24 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "84Λ͏্ͰͬͯΔ͜ͱ ɾ1FSM͔Β"84ΛૢΔ
ɾ3%4Θͣ.Z42-PO&$ ɾࢹ$MPVE8BUDIΛ͍ͬͯͳ͍ ɾ"VUP4DBMJOHࣗલͰߏங͍ͯ͠Δ ɾ&-#ͷঢ়ଶมԽΛࢹ 26 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. 1FSM͔Β"84ΛૢΔ ɾ"84$-*8SBQQFSΛར༻
w จࣈ௨Γ"84$-*ͷϥούʔ w "84$-*ͱಉ͡Α͏ͳ͍উखͰ͑Δ w SFTBXTFD EFTDSJCFJOTUBODFT ɾৄࡉງ͞Μͷ1SPHSBNNJOH"84XJUI1FSMͰ 27 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. .Z42-PO&$ ɾ3%4ʹൺͯϝϦοτ͕ଟ͍
w .)"Ͱ3%4ΑΓߴʹϑΣΠϧΦʔόʔͰ͖Δ w ετϨʔδΤϯδϯ͕ࣗ༝ʹબΔ 28 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. .)"WT3%4.VMUJ"; ɾো࣌ͷϑΣΠϧΦʔόʔ࣌ؒͷҧ͍
w .)"ඵఔɺ3%4ʙఔ ɾϝϯςφϯεੑ εέʔϧΞοϓεΩʔϚมߋ w ͲͪΒແఀࢭͰՄೳ ɾϚελΓସ͑࣌ͷϨϓϦΧܨ͗͠ w .)"৽Ϛελʹܨ͗ͯ͘͠ΕΔ 29 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ετϨʔδΤϯδϯ͕બΔ ɾ2.)BOEMFS4PDLFU͕͑Δ
w ࣾʹ࡞ऀ͕͍͕ͯ͋Ε͙͢ରԠͰ͖Δ w "84ͳΒ424%ZOBNP%#͋Δ͚Ͳ ɾ༰ྔޮͷ͍͍5PLV%#ͳͲબͰ͖Δ w JOOPECʹൺͯͷαΠζʹͳΔ ˞ w ϢʔβΞΫςΟϏςΟͷอ࣋ͳͲʹ༗༻ 30 ˞5PLV%#W&WFO#FUUFS$PNQSFTTJPO -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. $MPVE8BUDIͷΞϥʔϜ ɾ୯७ͳػೳ͔͠ఏڙ͞Εͯͳ͍ͷͰͬͯͳ͍
ɾαʔό͕ଟ͍ͱΞϥʔτരൃͯ͠͠·͏ w શ8FC͔ΒҰ੪ʹ$16ͷ௨͕དྷΔ ɾҰ࣌తʹ௨ࢭΊΔΈ͍ͨͳػೳ͕ݟͨΒͳ͍ w /BHJPTͷ4DIFEVMFE%PXOUJNF૬ 31 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ࢮ׆ࢹ/BHJPT ɾઃఆϑΝΠϧߏங࣌ʹࣗಈੜ͍ͯ͠Δ
ɾಛఆΫϥελͷΞϥʔτΛ·ͱΊΔΈΛ࣮ w $16ΞϥʔτΫϥελຖʹ௨͔͜͠ͳ͍ ɾ/BHJSB /"(*PT3FTUGVM"QJ w /BHJPTͷεςʔλεΛ3&45"1*ͰऔΕͯศར 32 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. $MPVE8BUDIͷάϥϑ ɾ͍উख͕Α͘ͳ͍ͷͰ͍ͬͯͳ͍
w िؒͷσʔλ͔͠ΈΕͳ͍ w άϥϑ͕ҰཡͰ͖ͳ͍ w /BNF5BHTͰϝτϦΫε୳ͤͳ͍ w ΞΧϯτ͕ෳ͋ΔͱΓସ͕͑໘ 33 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. αʔόϦιʔεࢹ ɾ$MPVE'PSFDBTU
w $16.FN%JTL%#ͷ214ͳͲ w ϫʔΧʔΛࢄॲཧͤͯ͞εέʔϧΞτͰ͖Δ w ඞཁʹԠͯ͡औಘ͢ΔϝτϦΫεΛ૿͢ͷ༰қ • 1PSUJOH.Z42-HSBQITGSPNQFSDPOBNPOJUPSJOHQMVHJOT • "EETFWFSBMHSBQITPGSFEJT 34 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. τϨϯυࢹ ɾ(SPXUI'PSFDBTU
w )551ͰΛ1045͢Δ͚ͩͰՄࢹԽͰ͖Δ w &-#ͷϦΫΤετɾΤϥʔ w %#ͷTIBSEຖͷϢʔβͷภΓ ɾ,JCBOB 'MVFOUE &MBTUJD4FBSDI w ۙͷΞΫηεϩάղੳʹͬͯΔ 36 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. (SPXUI'PSFDBTU ɾྫDMPVEXBUDIHGͰ&-#ΛՄࢹԽ
w ('ͷ͍ΔαʔόͰEBFNPOىಈ͢Δ͚ͩ w IUUQTHJUIVCDPNIJSPTFBXTDMPVEXBUDIHG 37 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ,JCBOBͰ݅ͷߜΓࠐΈ ɾಛఆͷ63-ͰߜΓࠐΈ
ɾ&$ͷಛఆλάͰߜΓࠐΈ ! ɾ&$ͷϝλσʔλͰߜΓࠐΈ 39 tagset_project: ff path: /user/* az: us-‐west-‐1b AND instance_type: m1.large -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ࢀߟ'MVFOUEͷૹ৴ଆ <source>
type tail_ex tag raw.acc.ff path /var/log/apache/ff.acc.%Y%m%d format ltsv </source> ! <match raw.acc.ff> type record_reformer remove_keys conn_status,num_ka,pid,user,ua,referer enable_ruby false output_tag reformed.acc.ff </match> ! <match reformed.acc.ff> type ec2_metadata output_tag acc.ff <record> hostname ${tagset_name} project ${tagset_project} instance_type ${instance_type} az ${availability_zone} </record> </match> 42 ϩάಡΈࠐΈ ෆཁϑΟʔϧυআ &$ͷϝλσʔλՃ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ࢀߟ'MVFOUEͷड৴ଆ <match
acc.**> type record_reformer output_tag reformed.${tag} path ${req} </match> ! <match reformed.acc.*> type rewrite remove_prefix reformed add_prefix es <rule> … </rule> </match> ! <match es.acc.*> type forest subtype elasticsearch remove_prefix es <template> logstash_format true </template> </match> 43 SFRϑΟʔϧυΛෳ ΫΤϦετϦϯάআڈ &MBTUJDTFBSDIʹอଘ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "VUP4DBMJOHЌ ɾಠࣗͷ࣮Ͱ͍ͬͯΔ
w ඞཁʹԠͯ͡ΧελϚΠζ͢Δͷ͕༰қ ɾରXFCαʔόͷΈ ɾ$16ͱϫʔΧʔͷϏδʔ͕ج४ 44 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "VUP4DBMJOHͷϙϦγʔ ɾΞΫςΟϏςΟ͕ଟ͍࣌ؒεέʔϧͤ͞ͳ͍
w ٸܹͳมԽʹαʔόՃ͕ؒʹ߹Θͳ͍ w ήʔϜຖʹϑΟʔόʔλΠϜ͕͋Γ༧ଌ͕͍͠ w ͔ͱ͍ͬͯᮢΊʹ͢Δͱແବ͕ଟ͍ ɾΞΫςΟϏςΟ͕͍࣌ؒʹΛݮΒ͢ʢಛʹਂʣ w ಥൃతʹΞΫςΟϏςΟ͕૿͑Δ͜ͱ·ͣͳ͍ 45 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "VUP4DBMJOHͷྫ 46
ϦΫΤετ Πϯελϯε $16༻ ͋ΔఔΩʔϓ ੵۃతʹࢭΊΔ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. &-#ͷঢ়ଶࢹ ɾ&-#ͷ*1ΞυϨεͷมԽΛ*3$ʹ௨
w &-#εέʔϧΞοϓͨ͠Γ͢Δͱ*1͕มΘΔʁ w ΞϥʔτͷݪҼΓ͚ʹར༻͍ͯ͠Δ w IUUQTHJUIVCDPNIJSPTFNPOJUPSFMCBEESFTT 47 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "84Ͱ࣮ݱͰ͖ͨΒ͍͍͜ͱ ɾଞͷ*BB4ͷ͍͍ͱ͜ͲΓ
ɾϦʔδϣϯؒ௨৴ͷ্ ɾ&-#ͷݟ͑ΔԽɾແ௨৴ ɾϚωδϝϯτίϯιʔϧͷϚϧνΞΧϯτ ɾαϙʔτʹؔͯ͠ 48 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ΄͔ͷ*BB4ͷ͍͍ͱ͜ͲΓ ɾͨͱ͑(PPHMF$PNQVUF&OHJOF͔Β
w ୯ҐͰͷ՝ۚ w ٸܹͳΞΫηε૿Ճʹ͑͏Δϩʔυόϥϯα w ϥΠϒϚΠάϨʔγϣϯ w ෳ7.ʹଓՄೳͳ1FSTJTUFOUσΟεΫ 49 ˞"84ΑΓ(PPHMF$PNQVUF&OHJOFΛબͼͨ͘ͳΔͷཧ༝ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Ϧʔδϣϯؒ௨৴ͷ্ ɾάϩʔόϧͳαʔϏεͰϦʔδϣϯؒ௨৴͍͖͍ͯͨ͠
w ݱঢ়+164Ͱશʹಠཱͨ͠ڥʹͳ͍ͬͯΔ w "QQϦʔδϣϯຖɺ%#ڞ௨Έ͍ͨʹ͍ͨ͠ ɾ*#.ͷTPGUMBZFS͕͜ͷ͋ͨΓ༏Ґͱͷ͜ͱ ˞ w ֤%$ͱ/8ڌ͕(CQTͷઐ༻ઢ w ωοτϫʔΫͷ༻ྉແྉ 50 ˞"84Ϧʔδϣϯؒ௨৴্ͱࠓޙͷΤίγεςϜ -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ϢʔβʹΑΔ&-#ͷཧ ɾιʔγϟϧήʔϜಥൃతͳΞΫηε૿͕ৗ
w ຖճϓϨΥʔϛϯάͪΐͬͱπϥΠ ɾ͏ͪΐͬͱϢʔβ͕ཧͰ͖ΔΑ͏ʹͳΔͱخ͍͠ w ΩϟύγςΟ֬ೝɺࣄલʹΩϟύγςΟՃ 51 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. &-#ͷڧ੍அ ɾ)551ͷϦΞϧλΠϜ௨৴Ͱෆศ
w ήʔϜνϟοτͳͲͰར༻͢Δέʔε͕૿͑ͯΔ w ແ௨৴ঢ়ଶ͕ଓ͘ͱڧ੍తʹଓ͕Εͯ͠·͏ w ࠷େͰ·Ͱ৳ͤΔͱ͍͏ϫα ˞ w ΦϓγϣϯͰઃఆͰ͖Δͱخ͍͠ 52 ※ https://forums.aws.amazon.com/thread.jspa?messageID=382182# -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Ϛωδϝϯτίϯιʔϧ ɾϚϧνΞΧϯτରԠͯ͠΄͍͠
w αʔϏε͕ͨ͘͞Μ͋ΔͱߦͬͨΓདྷͨΓ͕ͪ͠ w Ԧಓͳͷ$ISPNFͷϚϧνϢʔβػೳʁ w ผͷϒϥβΛىಈ͢Δͱ͍͏ख 53 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. "844VQQPSU ɾͪΐ͏Ͳ͍͍αϙʔτϓϥϯ͕ͳ͍
w ۓٸͳ࣭Λ·ΕʹૹΔఔ w &-#ͷϓϨΥʔϛϯά͍͍͕ͨ࣌͋Δ w ར༻ྉۚͷύʔηϯτ͔͔Δ ɾαϙʔτΦϯσϚϯυͰ w ༏ઌ͖ͷνέοτങ͑ΔΈ͍ͨͳΈ͕Α͍ʁ 54 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ͥΜͿ"84ͰΔϝϦοτ ɾγεςϜશମΛ"84্Ͱͷӡ༻ʹ࠷దԽͰ͖Δ
w ݱঢ়ΦϯϓϨͷγεςϜΛ"84Ͱӡ༻ͯ͠Δ ɾӡ༻໘Ͱͷ͕ݮΔ w αʔόͷނো Πϯελϯεো ͕ൺֱతগͳ͍ w ΩϟύγςΟϓϥϯχϯάָ͕ʹ 58 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ͥΜͿ"84ͰΔσϝϦοτ ɾ͚ࣗͨͪͩͰͲ͏ʹͰ͖ͳ͍ཁૉ͕૿͑Δ
w ϓϥοτϑΥʔϚʔͱͯ͠ͷઆ໌͕Ռͨͤͳ͍ ɾେنʹͳͬͯ͘ΔͱίετϝϦοτ͕ݟ͍ͩ͠ʹ͍͘ w ن͕େ͖͍ͱେྔʹόϧΫͰೖΕՄೳ w 3*͋ΓͰίετ༏ҐʹͳΒͳ͍ 59 -
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. ·ͱΊ ɾߟ͑ͯΈΔͱ"84Ҡߦ͢Δͷʹػೳతͳนগͳ͍
w ͦΕ͚ͩ"84ͷαʔϏε͕ߴ͍ ɾҰํͰɺۚમతͳ໘ͰҠߦϝϦοτ͕Έ͑ͳ͍ঢ়گ w ن͕େ͖͍ͱݟੵΓ͍͠Ұ໘͋Δ͕ w େ͖ͳϝϦοτ͕ͳ͍ͱ։ൃଆͷ֬อ͍͠ 60