Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ヘッドレスChromeでクローラを作った後の話

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.

 ヘッドレスChromeでクローラを作った後の話

Avatar for yujiosaka

yujiosaka

March 20, 2018
Tweet

More Decks by yujiosaka

Other Decks in Technology

Transcript

  1. Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ
 ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ

    ✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ੡ https://github.com/yujiosaka/headless-chrome-crawler
  2. ظ଴͍ͯͨ͠ϝϦοτ ✓ ඼࣭ͷ޲্ ✓ ։ൃྗͷ޲্ ✓ ϒϥϯυ޲্ ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞

    ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ Issue΍Βϝʔϧ΍ΒରԠ๩͍͠
  3. ར༻ͷϋʔυϧ ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ ✓ Headless Chrome + Crawler = Headless

    Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍ ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ
  4. ৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓

    ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍ ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ