Share
Embed
jdk-1812-mugt-session-latter-half-iqon-crawler-on-mesos
Japan Container Days v12.18 - DAY 2 [ 2W4 ]
Hashtags : #containerdaysjp #HallWEST ( #MUGT )
Mesos User Group Tokyo Community Session
IQON Crawler on Mesos
Tatsuro Mitsuno
December 05, 2018
Transcript
37 slides
-
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Japan Container
Days v12.18 - DAY 2 [ 2W4 ] Hashtags : #containerdaysjp #HallWEST ( #MUGT ) ZOZO Technologies, Inc. ։ൃ෦ SRE ςοΫϦʔυ ޫ ୡ࿕ *20/Ϋϩʔϥʔج൫ ʲ.FTPTϢʔβࣄྫʳ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϓϩϑΟʔϧ ϠϑʔגࣜձࣾʹͯϠϑΦΫ!ɾࠂͷ։ൃʹܞΘΔɻ2016͔
ΒגࣜձࣾVASILYɺ๏ਓ౷߹ʹ͍20184݄͔Βݱ৬ɻݱ ࡏSREΤϯδχΞͱͯ͠ɺʑαʔϏεͷ৴པੑ্ʹΊ ΔɻVASILY࣌ͷऔΓΈ͔Βԑ͋ͬͯMesos User Group TokyoʹࢀՃɻຊ൪ڥͰͷ࣮ફωλ୲ɻΟεΩʔͱ༿ר͕ ͖ɻѪͯ͠Δɻ ;0;05FDIOPMPHJFT *OD ։ൃ෦43&ςοΫϦʔυ ޫୡ࿕ Icon illustrated by YOSHI https://ja-jp.facebook.com/yoshi.yone.7 -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ;0;05FDIOPMPHJFT *OD
ZOZOグループのサービス運用・技術開発部門 -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ຊ࠷େڃͷϑΝογϣϯγϣοϐϯάαΠτΞϓϦ ˓
Ҏ্ͷγϣοϓɺ Ҏ্ͷϒϥϯυͷऔΓѻ͍ʢ ݄࣌ʣ ˓ৗ࣌ສҎ্ͷΞΠςϜͱຖฏۉ Ҏ্ͷ৽ண Λܝࡌ ˓ଈૹαʔϏεΪϑτϥοϐϯάαʔϏεπέ͍ͳͲ IUUQ[P[PKQ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ຊ࠷େڃͷϑΝογϣϯίʔσΟωʔτΞϓϦ ˓
ສμϯϩʔυಥഁɺίʔσΟωʔτߘ૯ສ݅Ҏ ্ʢͱʹ݄࣌ʣ ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ͕ར༻ՄೳͳશͯͷࠃʣͰμ ϯϩʔυ͕Մೳ ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔβʔʢ8&"3*45"ʣੜ IUUQTXFBSKQ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔεʔπ ˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛεϚʔτϑΥϯΧϝϥͰࡱ
Ө͢Δ͜ͱͰɺମܕσʔλΛܭଌ ˓ܭଌͨ͠ମܕσʔλɺॠ࣌ʹ%ϞσϧԽ͞Εɺ;0;0508/Ξ ϓϦʹอଘɻ%Ϟσϧ͋ΒΏΔ֯ʹಈ͔͢͜ͱ͕Ͱ͖ɺମܕΛ νΣοΫ͢Δ͜ͱ͕Մೳ IUUQ[P[PKQ[P[PTVJU -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔλΛͱʹɺҰਓͻͱΓͷମ ܕʹ߹ͬͨʮ͋ͳͨαΠζʯͷΞΠςϜ
˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔγοΫΞΠςϜΛఏڙ άϩʔόϧαΠτʮ;0;0DPNʯͰւ֎ల։ ˓ΞΠςϜ̩γϟπɺσχϜύϯπɺγϟπɺϏδωεεʔπɺ ωΫλΠɺϘʔμʔ̩γϟπɺକΫϧʔωοΫ̩γϟπͳͲ IUUQ[P[PKQQC -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷϏδωε EC
sites 1. ใ ू 3. ߪೖՄೳͳECαΠτૹ٬ 4. ܦ༝ച্ͷҰ෦ 2. Ӿཡ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. *20/ͷϏδωε EC
sites ॏཁ IQON Ϋϩʔϥ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. εέʔϧײʢ࣌ʣ online-shot:
Icons made by Vectors Market from www.flaticon.com is licensed by CC 3.0 BY dress: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY more than 300 EC sites per day more than 1,000,000 fashion items per day -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Application Infrastructure
RubyͰ࣮ ࢄฒྻ γεςϜԽ EC2 Ϋϩʔϥʔͷਐา ߋͳΔଟஈԽ XPSLFS # XPSLFS " &$*OTUBODF XPSLFS " XPSLFS $ XPSLFS % XPSLFS " -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Application Infrastructure
Ϋϩʔϥʔͷਐา ߋͳΔଟஈԽ XPSLFS # XPSLFS " &$*OTUBODF XPSLFS " XPSLFS $ XPSLFS % XPSLFS " ࣍ͳΔεςʔδ ʢ2016ʣ EC2 -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #
XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME Ϋϩʔϥʔͷཧ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #
XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME ՝ • ϫʔΧʔຖʹCPU, ϝϞϦͱ͍ͬͨඞཁͳϦιʔε͕ҟͳΔ • ͦͷ࣌ʑͰݸʑͷΠϯελϯεϦιʔεΛߟྀ͢Δඞཁ͕͋Δ ʢͨ͘͠ͳ͍ʣ • ϫʔΧʔͷϓϩηεΛίϯτϩʔϧϓϨʔϯ͔Βૢ࡞Մೳʹ͢Δඞཁ͕͋Δ ʢͨ͘͠ͳ͍ʣ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. XPSLFS #
XPSLFS " XPSLFS $ XPSLFS % XPSLFS " XPSLFS " XPSLFS # XPSLFS " XPSLFS $ XPSLFS % XPSLFS % XPSLFS % XPSLFS # XPSLFS $ XPSLFS % XPSLFS % TIME Ϋϩʔϥʔͷཧ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ىಈɾഇغ͕ ༰қͳശ
ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ $PNQVUFS $PNQVUFS Կ͔ૉఢͳநԽ XPSLFS " XPSLFS # $PNQVUFS XPSLFS $ XPSLFS % $PNQVUFS $PNQVUFS $PNQVUFS -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ىಈɾഇغ͕ ༰қͳശ
ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ ىಈɾഇغ͕ ༰қͳശ $PNQVUFS $PNQVUFS ίϯςφΦʔέετϨʔγϣϯ XPSLFS " XPSLFS # $PNQVUFS XPSLFS $ XPSLFS % $PNQVUFS $PNQVUFS $PNQVUFS -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT ܭࢉࢿݯ
ܭࢉࢿݯͷཧɾ ूɾநԽ ίϯςφͷঢ়گࢹ ΞϓϦέʔγϣϯ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ΫϩʔϥʔPO.FTPT ࢄγεςϜ༻
λεΫQueue Queue ࢹ Queueࢹ APIʹΑΔίϯςφૢ࡞ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. racing: Icons
made by Freepik from www.flaticon.com is licensed by CC 3.0 BY crawler operating time 67% down -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͳͥ.FTPTʁ shakehand,
home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿ୳͍ͯͨ͠ͷͱࢥ͕ϕετϚον shakehand,
home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY • MesosͷࢥɹɿࢄڥΛڊେͳܭࢉࢿݯͱͯ͠Έͳ͢ • ୳͍ͯͨ͠ͷ ɿϫʔΧʔΛ߹Α͘ஔ͢ΔͨΊͷܭࢉࢿݯ • ϫʔΧʔͷಛੑɿࢄɾඇಉظॲཧ ʹʼ CPUϝϞϦͱ͍ͬͨܭࢉࢿݯʹͷΈґଘ • MesosͷࢥγϯϓϧͰɺ͔ͭϕετϚονͩͬͨ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿγϯϓϧͳΫϥελ shakehand,
home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY Mesos Master Zookeeper Mesos Slave Framework (Optional) • ࠷খ3ཁૉ • ϑϨʔϜϫʔΫ͖ͳ͚ͩ • LinuxσΟετϦʹΑͬͯ apt/yumͰΠϯετʔϧՄೳ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ϙΠϯτͦͷɿ๛ͳυΩϡϝϯτ shakehand,
home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY • ৄࡉͳެࣜυΩϡϝϯτ • ΤʔδΣϯτͷύϥϝʔλ • Ξυόϯευͳ͍ํͱ۩ମతͳྫ • APIͷ༷ • ʮެࣜΛ୳ͤྑ͍ʯ ͱ͍͏҆৺ײ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͳͥ.FTPTʁ shakehand,
home: Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY open-book: Icons made by Zlatko Najdenovski< from www.flaticon.com is licensed by CC 3.0 BY -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͱ͜ΖͰ%$04ʁ ܭࢉࢿݯ
Mesos Marathon ʢDC/OS͕ࣗΛཧ͢ΔͨΊʣ DC/OS Marathon ʢΞϓϦέʔγϣϯ༻ʣ ΞϓϦέʔγϣϯ }1.ϑϨʔϜϫʔΫͷ͕ް͘ͳΔ 2.DC/OSͷརͰ͋ΔͦͷଞͷࢄγεςϜΛಈ͔͢༧ఆ͕ͳ͍ 3.DC/OS͕࣋ͭϩʔυόϥϯαɾDNS͕ෆཁ γϯϓϧʹѻ͏ͨΊɺDC/OSෆ࠾༻ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ·ͱΊ •IQONΫϩʔϥʔج൫Mesos͕ΘΕ͍ͯΔ
• MesosʹΑͬͯϫʔΧʔͷ͕ޮతʹߦ͑Δ༷ʹ • ࢄॲཧͰ͖ΔΞϓϦέʔγϣϯͰ͋ΕMesos͕ΑΓྑ͍બࢶͱͳΔ •Ϋϩʔϧͷऴ͕ྃ࣌ؒ1/3ʹॖ •τϥϒϧ࣌ͷλεΫ࠶ஔ͕ࣗಈͰߦΘΕΔͨΊγεςϜͷՄ༻ੑ্ • ͜ΕΛར༻ͯ͠AWS SpotInstanceͳͲΛΈ߹ΘͤΔͱίετݮՄೳ •ඞͣ͠DC/OSඞཁͰͳ͍ͷͰɺέʔεόΠέʔεͰબ ✖ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͬͱΓ͍ͨ Docker
/ Apache Mesos / Marathon ʹΑΔ3ഒ͍IQONΫϩʔϥʔͷߏங https://techblog.zozo.com/entry/iqon-crawler-by-docker-and-mesos-and-marathon Apache Mesos with Amazon EC2 SpotFleet https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet