1日あたり数百万商品をクロールする 大規模クローラーの裏側 / How IQON crawler works

Transcript

1. © 2017 VASILY,Inc. ೔͋ͨΓ਺ඦສ঎඼ΛΫϩʔϧ͢Δ
   େن໛Ϋϩʔϥʔͷཪଆ 4QFFF
   $BGF
   .FFUVQ
   
   Ԙ㟒݈߂
2. © 2017 VASILY,Inc. ࣗݾ঺հ ⾣ Ԙ㟒݈߂
   ⾣ 
   7"4*-:৽ଔೖࣾ
   ⾣ όοΫΤϯυΤϯδχΞ
   ⾣ 3VCZ
   ⾣ (PPHMF
   #JH2VFSZ
   ⾣ "QBDIF
   4PMS
   ⾣ &NCVML
   ⾣ %JHEBH
   ▶ $ crontab -l
 0 0 7 8 * /bin/increment_age  ฐٕࣾज़ސ໰
   .BU[ࢯ
3. © 2017 VASILY,Inc. ໊ࣾ
   גࣜձࣾ7"4*-: ϰΝγϦʔ
   
   7"4*-: *OD
   ઃཱ
   ೥݄
   ॴࡏ஍
   ౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ
   '
   ैۀһ
   ໊
   ࢿຊۚ
   ԯ
 ୅දऔక໾
   ۚࢁ
   ༟थ
 औక໾ࠓଜ
   խ޾
   
   ઍ༿
   େี
 ओཁגओ
   
   άϩʔϏεϕϯνϟʔΩϟϐλϧ
   ҏ౻஧ςΫϊϩδʔϕϯνϟʔζ
   (.0ϕϯνϟʔύʔτφʔζ
   ,%%*גࣜձࣾ
   גࣜձࣾߨஊࣾ
4. © 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔Βͷ΂ສ఺Λ௒͑Δ঎඼Λܝࡌ ݄ؒສਓҎ্͕ར༻͢Δ೔ຊ࠷େڃͷϑΝογϣϯαΠτ
5. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS
   
   .FTPT
   
   .BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
6. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS
   
   .FTPT
   
   .BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
7. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ఏܞαΠτ਺
   Ҏ্
   ⾣ ৗ࣌ߪങՄೳ఺਺ɿສ఺Ҏ্
8. © 2017 VASILY,Inc. *20/ͷΫϩʔϥʔ  ⾣ ໿߲໨ͷ঎඼৘ใΛऔಘ ঎඼ࣸਅ Ձ֨ ࡏݿ਺ ϒϥϯυ ঎඼໊
9. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
   ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
   ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
   ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
   ⾣ εέʔϥϒϧͳΠϯϑϥ
   ⾣ %PDLFS
   
   .FTPT
   
   .BSBUIPOʹΑΔΦʔτεέʔϧ
   ⾣ ·ͱΊ
10. © 2017 VASILY,Inc. ෼ࢄΫϩʔϦϯά ⾣ ର৅ϖʔδ͕๲େͳͷͰ෼ࢄॲཧ
    ⾣ ෼ࢄΫϩʔϧ༻ͷϑϨʔϜϫʔΫ͸ͳ͍
    ⾣ 3VCZΛ༻͍ϑϧεΫϥονͰ࣮૷
     4DSBQZ
    EPFTO`U
    QSPWJEF
    BOZ
    CVJMUJO
    GBDJMJUZ
    GPS
    SVOOJOH
    DSBXMT
    JO
    B
    EJTUSJCVUF
    NVMUJTFSWFS
    NBOOFS
    IUUQTEPDTDSBQZPSHFOMBUFTUUPQJDTQSBDUJDFTIUNMEJTUSJCVUFEDSBXMT
11. © 2017 VASILY,Inc. UBTL
    RVFVFΛհͨ͠෼ࢄɾඇಉظॲཧ ⾣ 424ΛUBTL
    RVFVFͱͯ͠࢖༻
    ⾣ UBTLΛ࣮ߦ͢ΔϓϩηεʢXPSLFSʣಉ࢜͸ૄ݁߹
    ⾣ ඇಉظॲཧϥΠϒϥϦͱͯ͠ɺ4IPSZVLFOΛ࢖༻
     XPSLFS XPSLFS FORVFVF EFRVFVF 424
12. © 2017 VASILY,Inc. 4IPSZVLFO  ⾣ 3VCZͰॻ͔ΕͨඇಉظॲཧϑϨʔϜϫʔΫ
    ⾣ ෳ਺Ωϡʔͷ؅ཧػೳ
    ⾣ ϚϧνεϨου class HelloWorker include Shoryuken::Worker shoryuken_options queue: 'hello', auto_delete: true def perform(sqs_msg, name) puts "Hello, #{name}" end end HelloWorker.perform_async('joe') BTZOD
    DBMM
13. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
14. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
15. © 2017 VASILY,Inc. $MPVE8BUDI
    
    -BNCEB ⾣ ఆظతʹ$MPVE8BUDI
    &WFOU͕ൃՐ͠ɺ-BNCEBΛىಈ
    ⾣ -BNCEB͕424ʹλεΫΛ౤ೖ͢Δ͜ͱͰɺΫϩʔϧ։࢝ $MPVE8BUDI 4/4 -BNCEB 424 JOWPLF FORVFVF 
16. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
17. © 2017 VASILY,Inc. TUBSUDSBXMFS
    XPSLFS ⾣ 3%4͔Βͦͷ೔ͷΫϩʔϧର৅αΠτ৘ใΛऔಘ
    ⾣ ͦΕΒͷαΠτΛΫϩʔϧ͢ΔͨΊͷλεΫΛ౤ೖ TUBSUDSBXMFS 424 3%4 
    Ϋϩʔϧର৅αΠτ৘ใऔಘ 
    FORVFVF º
    &$αΠτͷ਺
18. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
19. © 2017 VASILY,Inc. MJTUQBHFFORVFVF
    XPSLFS ⾣ &$αΠτͷϖʔδૹΓ෦෼ΛεΫϨΠϐϯά
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ MJTUQBHFFORVFVF 424 
    ϖʔδૹΓ෦෼ղੳ 
    FORVFVF º
    ϖʔδͷ਺ &$
    TJUF IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF
    IUUQTFYBNQMFDPNJUFNT QBHF
20. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
21. © 2017 VASILY,Inc. JUFNQBHFFORVFVF
    XPSLFS ⾣ ঎඼Ϧετϖʔδ͔Β঎඼ৄࡉϖʔδ΁ͷ63-Λղੳ
    ⾣ શϖʔδͷ63-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ JUFNQBHFFORVFVF 424 
    Ϧετϖʔδˠৄࡉϖʔδͷ63-Λऔಘ 
    FORVFVF º
    ϖʔδͷ਺ &$
    TJUF IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT
    IUUQTFYBNQMFDPNJUFNT
22. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
23. © 2017 VASILY,Inc. JUFNQBHFEPXOMPBE
    XPSLFS ⾣ ঎඼ৄࡉϖʔδ͔Β)5.-Λμ΢ϯϩʔυ
    ⾣ μ΢ϯϩʔυִؒͷௐ੔ͷͨΊʹ3FEJTʹ෼ࢄϩοΫʢޙड़ʣΛ࣮ݱ
    ⾣ )5.-Λղੳ͢ΔͨΊͷλεΫΛ౤ೖ JUFNQBHFFORVFVF 424 
    ৄࡉϖʔδͷ)5.-Λऔಘ 
    FORVFVF &$
    TJUF <!DOCTYPE> <HTML><HEAD><TITLE>トップス... 
    ϩοΫऔಘ
24. © 2017 VASILY,Inc. શମߏ੒ਤ  TUBSUDSBXMFS MJTUQBHFFORVFVF JUFNQBHFFORVFVF JUFNQBHFEPXOMPBE JUFNQBHFQBSTF ⾣ ΫϩʔϧॲཧΛෳ਺ݸͷλεΫʹࡉ෼Խ $MPVE8BUDI
    
    -BNCEB
25. © 2017 VASILY,Inc. JUFNQBHFQBSTF
    XPSLFS ⾣ 91BUIɾਖ਼نදݱΛ࢖͍)5.-Λύʔε
    ⾣ ύʔεઃఆʢ91BUIɾਖ਼نදݱ͸XFCπʔϧͰೖߘ
    ⾣ ύʔε݁ՌΛ%#ʹॻ͖ࠐΉλεΫΛ౤ೖ JUFNQBHFQBSTF 424 
    ύʔεઃఆΛऔಘ 
    FORVFVF { "title": "トップス", "price": 9800, 3%4 ύʔεઃఆΛೖߘ 
26. © 2017 VASILY,Inc. ͜ΕҎ߱ͷॲཧ ⾣ ࣌ؒͷ౎߹Ͱࠓճ͸ׂѪ
    
    ⾣ Ϋϩʔϧ݁ՌΛ%#ʹॻ͖ࠐΈ
    ⾣ ը૾ॲཧ
    ⾣ ಁաॲཧ
    ⾣ ୅ද৭நग़
    ⾣ ঎඼ΧςΰϦʔࣗಈ෼ྨ 
27. © 2017 VASILY,Inc. μ΢ϯϩʔυִؒ  ⾣ 3FEJTͰ෼ࢄϩοΫΛ࣮ݱ͠ɺμ΢ϯϩʔυִؒΛௐ੔ IUUQTSFEJTJPUPQJDTEJTUMPDL EPXOMPBE
    XPSLFS
    " EPXOMPBE
    XPSLFS
    # HFU@MPDL
    
    
    TVDDFTT MPDLFE HFU@MPDL
    
    
    GBJM EPXOMPBE HFU@MPDL
    
    
    GBJM FYQJSF HFU@MPDL
    
    
    TVDDFTT
28. © 2017 VASILY,Inc. จࣈίʔυ໰୊  ⾣ NFUB
    DIBSTFU͸࢖༻͍ͯ͠ͳ͍
    ⾣ ,DPOWʢOLGϥούʔʣͷจࣈίʔυࣗಈਪଌػೳΛར༻
    ▶ ::Kconv.toutf8(str)͚ͩͰ0,
29. © 2017 VASILY,Inc. 41" 4JOHMF
    1BHF
    "QQMJDBUJPO ͷରԠ  ⾣ 41"͸)5.-ʹ͸΄ͱΜͲͷ৘ใ͕ͳ͍
    ⾣ 1IBOUPN+4Λ࢖ͬͨ1SPYZΛհ͢Δ
    ⾣ PO-PBEΠϕϯτൃՐޙͷ৘ใΛऔಘ EPXOMPBE
    XPSLFS &$
    TJUF EPXOMPBE
30. © 2017 VASILY,Inc. 424ͷαΠζ੍ݶ  ⾣ 424ʹ͸,#ҎԼͷςΩετσʔλ͔֨͠ೲͰ͖ͳ͍
    ⾣ Ұ෦ͷϖʔδͷ)5.-͸͜ΕΛ௒ա
    ⾣ )UNM$PNQSFTTPS
    ;MJC
    #BTFͰ)5.-Λѹॖ
    ⾣ ʙఔ౓ʹѹॖ Base64.encode64(
 Zlib::Deflate.deflate(
 HtmlCompressor::Compressor.new.compress(html)
 )
 )
31. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS
    
    .FTPT
    
    .BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ
32. © 2017 VASILY,Inc. Πϯϑϥߏ੒ਤ  &$
    TQPU
    qFFU
    JOTUBODFT XPSLFST
    JO
    DPOUBJOFS %FQMPZ
    $POUBJOFS 424 -BNCEB $MPVE8BUDI 8BUDI
    .FUSJDT "VUP4MBDF FORVFVF
    
    EFRVFVF
33. © 2017 VASILY,Inc. "QBDIF
    .FTPT
    
    .BSBUIPO  ⾣ "QBDIF
    .FTPT
    ⾣ "
    EJTUSJCVUFE
    TZTUFNT
    LFSOFM
    ⾣ ෳ਺ϚγϯΛͭͷܭࢉػϓʔϧͱͯ͠ந৅Խ
    ⾣ .BSBUIPO
    ⾣ .FTPT্Ͱಈ࡞͢ΔίϯςφΦʔέετϨʔγϣϯπʔϧ
    ⾣ .FTPTͷλεΫΛσʔϞϯԽ
34. © 2017 VASILY,Inc. EPXOMPBE
    XPSLFSͷՔಇ཰  ίϯςφ਺ ॲཧ଎౓ ⾣ EPXOMPBE
    XPSLFSʹ͸ϩοΫ͕͋ΔͨΊɺ޻෉͕ඞཁ XJUI
    ޻෉ XJUIPVU
    ޻෉
35. © 2017 VASILY,Inc. EPXOMPBEΩϡʔͷଟॏԽ  ⾣ EPXOMPBEΩϡʔɾ෼ࢄϩοΫ͸&$αΠτຖʹಠཱ EPXOMPBE
    XPSLFS EPXOMPBE
    XPSLFS ϥϯμϜʹEFRVFVF αΠτʹରԠͨ͠
    ෼ࢄϩοΫΛऔಘ
36. © 2017 VASILY,Inc. ΦʔτεέʔϧͷͨΊͷϝτϦΫε  424 -BNCEB $MPVE8BUDI
    &WFOU ⾣ EPXOMPBEΩϡʔ͸ະॲཧͷλεΫͷ਺Λ؂ࢹͯ͠΋ҙຯ͕ͳ͍
    ⾣ ͦͷ୅ΘΓʹɺະॲཧλεΫ
    
    ͷΩϡʔͷݸ਺Λ؂ࢹ
    ⾣ $MPVE8BUDIͰ͸؂ࢹͰ͖ͳ͍
    ⾣ -BNCEBͰ424ͷ"1*Λୟ͘ JOWPLF HFU
    UIF
    OVNCFS
    PG
    
    OPOFNQUZ
    EPXOMPBE
    RVFVFT DIBOHF
    UIF
    OVNCFS
    PG
    DPOUBJOFST
37. © 2017 VASILY,Inc. 'PS
    NPSF
    JOGPSNBUJPO  ⾣ %PDLFS
    
    "QBDIF
    .FTPT
    
    .BSBUIPO
    ʹΑΔഒ଎͍*20/Ϋϩʔϥʔͷߏங
    ⾣ IUUQUFDIWBTJMZKQFOUSZJRPODSBXMFSCZEPDLFSBOENFTPTBOENBSBUIPO
    ⾣ "QBDIF
    .FTPT
    
    .BSBUIPO
    Λຊ൪Ͱӡ༻͢ΔͨΊͷͭͷ5JQT
    ⾣ IUUQUFDIWBTJMZKQFOUSZBQBDIFNFTPTBOENBSBUIPOUJQT
    ⾣ 1SPEVDUJPO
    EFQMPZNFOU
    PG
    UIF
    %PDLFS
    DPOUBJOFS
    XJUI
    .BSBUIPO
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVQSPEVDUJPOEFQMPZNFOUPGUIFEPDLFSDPOUBJOFSXJUINBSBUIPO
    ⾣ "QBDIF
    .FTPT
    XJUI
    "NB[PO
    &$
    4QPU'MFFU
    ⾣ IUUQTTQFBLFSEFDLDPNLPUBUTVBQBDIFNFTPTXJUIBNB[POFDTQPUqFFU
38. © 2017 VASILY,Inc. ໨࣍  ⾣ *20/ͷΫϩʔϥʔʹ͍ͭͯ
    ⾣ ن໛ɾऔಘ͍ͯ͠Δ৘ใ
    ⾣ ෼ࢄΫϩʔϦϯάΛ࣮ݱ͢Δཁૉٕज़
    ⾣ 424ͱ4IPSZVLFOΛ༻͍ͨඇಉظॲཧ
    ⾣ εέʔϥϒϧͳΠϯϑϥ
    ⾣ %PDLFS
    
    .FTPT
    
    .BSBUIPOʹΑΔΦʔτεέʔϧ
    ⾣ ·ͱΊ
39. © 2017 VASILY,Inc. ·ͱΊ  ⾣ *20/ͷΫϩʔϥʔ͸ຖ೔਺ඦສ঎඼ΛΫϩʔϧ͍ͯ͠Δ
    ⾣ େن໛ͳ෼ࢄΫϩʔϥʔΛ3VCZͰϑϧεΫϥονͰ࣮૷
    ⾣ ඇಉظॲཧΛ׆༻ͨ͠ॊೈͳΞϓϦέʔγϣϯ
    ⾣ εέʔϥϒϧͳΠϯϑϥͷ্ͰɺεϐʔυΞοϓˍඅ༻࡟ݮ
40. © 2017 VASILY,Inc. 8FSF
    )JSJOH  IUUQTWBTJMZKQSFDSVJU