ヘッドレスChromeでクローラを作った後の話

Transcript

1. ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ”࿩ Roppongi.js #1 Yuji Isobe
2. Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at
 
 https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
3. ΫϩʔϥΛ࡞ͬͨ࣌ͷ࿩ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana
4. ϔουϨεChromeͱ͸ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓ ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ ✓ ओͳ༻్͸ςετࣗಈԽͱΫϩʔϥ
5. Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ
 ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ ✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ੡ https://github.com/yujiosaka/headless-chrome-crawler
6. ࠓ೔͸࡞ͬͨ”ޙͷ”࿩
7. GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752
8. GitHub Trending Repositories ͜ͷลΛ2-3೔ؒ ͏Ζ͍ͭͯͨ https://github.com/trending
9. Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000
10. ເ͕׎ͬͨ
11. ظ଴͍ͯͨ͠ϝϦοτ ✓ ඼࣭ͷ޲্ ✓ ։ൃྗͷ޲্ ✓ ϒϥϯυ޲্ ✓ ࢓ࣄʹͭͳ͕Δ ୭͔࢓ࣄ͘ΕΖ͍ͩ͘͞ ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ Issue΍Βϝʔϧ΍ΒରԠ๩͍͠
12. ࢓ࣄதͣͬͱχϠχϠͰ͖Δ
13. ຊ୊ ଟ͘ͷਓͷ໨ʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λ੔ཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰ࿩൒෼ʹฉ͍͍ͯͩ͘͞
14. λʔήοτͷ޿͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱ͸ͳ͍ ✓ ʰ͸ͯϒϗοτΤϯτϦʔʱΑΓ
 ʰHacker News Top Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
 
 
 

15. ར༻ͷϋʔυϧ ✓ Ұ໨ݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ෼͔Δ ✓ Headless Chrome + Crawler = Headless Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ΋෼͔Βͳ͍ਓ޲͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ͸࠷ॳͷ਺ߦ͔͠ಡ·ͳ͍ ✓ ը૾΍ϩΰͰ΋ʮ؆୯͞ʯ͸ΞϐʔϧͰ͖Δ
16. ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ͸ examples ͩͬͨ ✓ πʔϧ͸ϑϧ׆༻͢Δ ✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
17. ৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕Ұ໨Ͱ෼͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓ ࠷ޙͷίϛοτ͕൒೥લͷϓϩδΣΫτͱ͔࢖͍ͨ͘ͳ͍ ✓ ͠͹Β͘αϘͬͯͯ΋ɺৗʹίϛοτͰ͖Δ
18. Happy Niya-niya Hacking!