Upgrade to Pro — share decks privately, control downloads, hide ads and more …

高速なBitTorrent DHTクローラ

高速なBitTorrent DHTクローラ

Keichi Takahashi

June 09, 2016
Tweet

More Decks by Keichi Takahashi

Other Decks in Programming

Transcript

  1. #JU5PSSFOU w ੈք࠷େͷ11ϑΝΠϧڞ༗ιϑτ΢ΣΞ ‣ ৗ࣌ສਓʙສਓͷΦϯϥΠϯϢʔβ ೥  ‣ ԯສਓͷΞΫςΟϒϢʔβ ೥

     w ༷ʑͳԠ༻ྫ ‣ #JU5PSSFOU4ZOD 11൛%SPQCPY  ‣ ΦϯϥΠϯήʔϜͷΞοϓσʔτͷ഑৴ $"1$0. #MJ[[BSE ʜ  ‣ Ϋϥελʹ͓͚ΔΞϓϦέʔγϣϯͷσϓϩΠπʔϧɾ
 ڞ༗ϑΝΠϧγεςϜͱͯ͠ 
  2. %JTUSJCVUFE)BTI5BCMF %)5 w )BTI5BCMF ‣ Ωʔ͔Βޮ཰తʹ஋ΛݕࡧͰ͖Δσʔλߏ଄ w %JTUSJCVUFE)BTI5BCMF ‣ σʔλΛෳ਺ͷίϯϐϡʔλ্ʹ෼ࢄ͢Δ)BTI5BCMF

    w ैདྷͷ#JU5PSSFOUͷऑ఺ ‣ 5SBDLFS͕ఀࢭ͢ΔͱɺϐΞͷ৘ใͷ୳ࡧ͕Ͱ͖ͳ͘ͳΓɺ
 μ΢ϯϩʔυɾΞοϓϩʔυෆೳʹͳΔ ‣ ϐΞ਺͕૿Ճ͢Δͱ5SBDLFS͕ϘτϧωοΫʹͳΓ͏Δ w #JU5PSSFOU.BJOMJOF%)5 ‣ ϐΞͷ৘ใΛ%)5ʹอଘ͢Δ͜ͱͰɺ଱ো֐ੑͱεέʔϥϏϦςΟ Λ޲্ 
  3. #JU5PSSFOU.BJOMJOF%)5ͷ࢓૊Έ w ,BEFNMJBΛϕʔεʹͨ͠%)5 ‣ CJUͷΞυϨεۭؒ ‣ 903ܭྔͱ෼໦ͰΞυϨεۭؒΛߏ଄Խ w %)5ϊʔυؒ͸,31$ͱ͍͏ϓϩτίϧͰ௨৴ ‣

    6%1্ͷඇಉظతͳϓϩτίϧ w QJOH pOE@OPEF HFU@QFFS BOOOPVODF@QFFSͷछͷϝοηʔδ ‣ QJOH JE ϊʔυͷੜࢮΛ֬ೝ ‣ pOE@OPEF JE UBSHFU UBSHFUʹۙͦ͏ͳϊʔυΛऔಘ ‣ HFU@QFFS JE JOGPIBTI JOGPIBTIΛ͍࣋ͬͯͦ͏ͳϐΞΛऔಘ ‣ BOOPVDF@QFFS JE JNQMJFE@QPSU JOGP@IBTI QPSU UPLFO 
 ϐΞͷμ΢ϯϩʔυΛΞφ΢ϯε 
  4. #JU5PSSFOU%)5ͷΫϩʔϥ w ͳͥͭ͘Δ͔ ‣ #JU5PSSFOUͱ%)5ͷཧղΛਂΊΔ ‣ Ϋϩʔϥͷઃܭɾ࣮૷ٕज़Λशಘ͢Δ ‣ Ԡ༻ 

    #JU5PSSFOUωοτϫʔΫͷ؂ࢹɾ෼ੳ  ྲྀ௨͍ͯ͠Δ5PSSFOU৘ใͷऩू ‣ ָ͍͔͠Β w ͳʹΛͭ͘Δ͔ ‣ %)5ʹࢀՃ͍ͯ͠ΔϊʔυΛҶͮΔࣜʹ୳ࡧ͍ͯ͘͠Ϋϩʔϥ ‣ ͨͩ࡞Δ͚ͩͰ͸໘ന͘ͳ͍ͷͰɺΫϩʔϦϯά଎౓ΛۃΊΔ 
  5. ઌߦݚڀ  1 DHTの迅速で膨大なモニタリング : 24時間以内に 1000万ノードの情報をクローリングする PacSec Tokyo November

    2011 In this presentation, we present our high-speed DHT Crawler to monitor 10 millions of nodes in 24 hours ! 情報通信研究機構 ネットエージェント株式会社 25 Rapid crawling: 24 hours to reach 10000000 nodes ! node 0 2000000 4000000 6000000 8000000 10000000 12000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 hour diff 10000 100000 1000000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 PacSec 2011 ·ͣ͸͜ΕΛ໨ඪʹ
  6. "SBDIOF., w ࣌ؒͰສϊʔυΛΫϩʔϧ ϫʔΧ  w ϐʔΫੑೳ͸ϊʔυඵϫʔΧ w OPEFKTͰૉ௚ʹ࣮૷ ಛผʹ࠷దԽͳͲ͸͍ͯ͠ͳ͍

     w ϫʔΧΛ૿΍͢͜ͱͰεέʔϧΞ΢τՄೳ w %JHJUBM0DFBO 714ϓϩόΠμ ্ʹߏஙͨ͠ΫϥελͰಈ࡞ ‣ ϫʔΧ ίΞ.#  7. ‣ δϣϒΩϡʔ ίΞ.#  7. ‣ σʔλϕʔε ίΞ(#  7. w ֤ϫʔΧͰ͸ฒྻͰಈ࡞  ^IPVS IUUQTHJUIVCDPNLFJDIJBSBDIOFNL
  7. "SBDIOF., w ෼ͰສϊʔυΛΫϩʔϧ ‣ %JHJUBM0DFBOͷίΞ(#44%ͷ7.୆্Ͱͷੑೳ w ϐʔΫੑೳ͸ສϊʔυඵ w ݱࡏ ͨͿΜ

    ੈք࠷଎ w ϊʔυͰͷ࣮ߦΛલఏͱͯ͠ઃܭ w ৭ʑ࠷దԽΛࢪͯ͋͠Δ  IUUQTHJUIVCDPNLFJDIJBSBDIOFNL ஫/"5഑ԼͰ࣮ߦ͢Δͱ
 /"5ςʔϒϧ͕ॠ࣌ʹᷓΕΔ
  8. "SBDIOF.,ͷ࢓૊Έ  pOE@OPEF
 ϦΫΤετߏங ૹ৴εϨου ϦΫΤετૹ৴ pOE@OPEF
 ϦϓϥΠͷύʔε ड৴εϨου ϦϓϥΠड৴

    ϊʔυΛ
 σΩϡʔ ཚ਺ιʔε 6%1ιέοτ ϊʔυΛ
 ΤϯΩϡʔ ΠϯϝϞϦ%# ϊʔυͷΩϡʔ
  9. ฒྻΩϡʔͷ࠷దԽ w Ωϡʔͷ࣮૷ ‣ ૒ํ޲࿈݁Ϧετ  ཁૉΛḷΔࡍʹϝϞϦΞΫηε͕ඈͼඈͼʹͳΔ ‣ ϦϯάόοϑΝ 

    ϝϞϦΞΫηε͕γʔέϯγϟϧʹͳΓΩϟογϡώοτ޲্  ݻఆ௕ͳͷͰࣄલʹϝϞϦΛ֬อ͓͚ͯ͠Δ w ഉଞ੍ޚ ‣ ϛϡʔςοΫεͰΩϡʔશମΛϩοΫ  ϩοΫɾΞϯϩοΫͷίετ͕ߴ͍ ‣ ΞτϛοΫ໋ྩΛ࢖ͬͨϩοΫϑϦʔͳΩϡʔ 
  10. Ϋϩʔϧ݁Ռͷอଘɾ໰͍߹Θͤ w Ϋϩʔϧͨ͠ϊʔυͷ৘ใΛอଘ͠ͳ͚Ε͹͍͚ͳ͍ w ϥϯμϜϦʔυɾϥΠτͷϨΠςϯγΛ࠷খԽ͍ͨ͠ ‣ ελϯυΞϩʔϯ%#  %#ͱͷ௨৴ͷϨΠςϯγ͕େ͖͗͢Δ ‣

    Τϯϕου,74  #PMU%#ͱ-FWFM%#Ͱࢼͨ݁͠ՌɺϨΠςϯγ͕େ͖͗͢Δ ‣ ΠϯϝϞϦ%#  ࣮ࡍʹ͸ͨͩͷHPඪ४ͷNBQ  ࠷଎͕ͩҰ੾ӬଓԽ͠ͳ͍  Ωϟογϡͱͯ͠࢖͍ɺఆظతʹ42-JUFͳͲʹॻ͘ͷ͕ݱ࣮త 
  11. ͦͷଞͷࡉ͔͍࠷దԽ w CFODPEFͷγϦΞϥΠβɾσγϦΞϥΠβ ‣ طଘϥΠϒϥϦ͸ϦϑϨΫγϣϯͳͲΛ࢖͍ͬͯΔͨΊ஗͗͢Δ ‣ γϦΞϥΠβ͸pOE@OPEFϦΫΤετͷΈʹରԠ ‣ σγϦΞϥΠβ͸pOE@OPEFϦΫΤετͷϦεϙϯεͷΈʹରԠ w

    ϝϞϦΞϩέʔγϣϯͷ࡟ݮ ‣ ՄೳͳݶΓࣄલʹϝϞϦ֬อͯ͠࢖͍ճ͢ w ϊʔυʹෳ਺ճpOE@OPEFϦΫΤετΛૹ৴ ‣ ֤ϊʔυ͔Βϊʔυͷ৘ใΛͰ͖Δ͚ͩଟ͘Ҿ͖ग़͢ w (PݴޠͰ࣮૷ ‣ OPEFKT଎͍ͱ͸͍͑ɺػցޠʹίϯύΠϧ͢Δݴޠͷํ͕଎͍ 
  12. ·ͱΊ w #JU5PSSFOU%)5͸଱ো֐ੑͱεέʔϥϏϦςΟΛ޲্͢ΔͨΊͷ࢓૊Έ w %)5ͷΫϩʔϥΛछྨ։ൃ ‣ "SBDIOF.,  ฒྻ෼ࢄܕͷΫϩʔϥ 

    ࣌ؒͰສϊʔυΛൃݟ ‣ "SBDIOF.,  ୯ҰϊʔυͰͷ࣮ߦ͢ΔΫϩʔϥ  ෼ͰສϊʔυΛൃݟ w কདྷ՝୊ ‣ ͞ΒͳΔ࠷దԽ ‣ UPSSFOUϝλ৘ใͷऩू 
  13. ࢀߟจݙ w #&1 ‣ IUUQXXXCJUUPSSFOUPSHCFQTCFQ@IUNM w ͳͥͳʹ5PSSFOU ‣ IUUQTXXXHJUCPPLDPNCPPLLZPSPIJSPEPD@IFUJNBUPSSFOUEFUBJMT w

    %)5ͷਝ଎Ͱ๲େͳϞχλϦϯά࣌ؒҎ಺ʹສϊʔυͷ৘ใΛΫ ϩʔϦϯά͢Δ ‣ IUUQTQBDTFDKQQTK1BD4FD@.BTTJWF.POJUPSJOH@KQQEG w %)5᠓઎ᗡ஬ ‣ IUUQFOHJZDPNUFDI w ,BEFNMJB ‣ IUUQXXXTIVEPOFUBSUJDMF,BEFNMJBTIVEP,BEFNMJBQEG