Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
ヘッドレスChromeでクローラを作った後の話
Search
yujiosaka
March 20, 2018
Technology
770
3
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
ヘッドレスChromeでクローラを作った後の話
yujiosaka
March 20, 2018
Transcript
18 slides
ϔουϨεChromeͰ ΫϩʔϥΛ࡞ͬͨ”ޙͷ” Roppongi.js #1 Yuji Isobe
Yuji Isobe @yujiosaka ϓϩδΣΫτϚωʔδϟʔ at https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
ΫϩʔϥΛ࡞ͬͨ࣌ͷ https://speakerdeck.com/yujiosaka/an-gazui-chu-nihetudoresuchromedekurorazuo-tuta-shi-ninannekana
ϔουϨεChromeͱ ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ දతͳϔουϨεϒϥβͱ͍͑PhantomJS ✓ ߴͰ҆ఆͯ͠ಈ࡞͢Δ ✓
ඪ४ͷରԠ͕ૣ͍ʢES2017Async-Await͕͑Δʣ ✓ ओͳ༻్ςετࣗಈԽͱΫϩʔϥ
Headless Chrome Crawler ✓ ϔουϨεChromeͰΫϩʔϥ ✓ ࢄڥͰಈ࡞͢Δ ✓ ਂ͞༏ઌ୳ࡧʢDFSʣͱ ෯༏ઌ୳ࡧʢBFSʣΛαϙʔτ
✓ robots.txt, sitemap.xmlʹै͏ ✓ Puppeteerʹґଘ ✓ Node.jsʢJavaScriptʣ https://github.com/yujiosaka/headless-chrome-crawler
ࠓ࡞ͬͨ”ޙͷ”
GitHub > Insights > Traffic https://twitter.com/yujiosaka/status/967316514322890752
GitHub Trending Repositories ͜ͷลΛ2-3ؒ ͏Ζ͍ͭͯͨ https://github.com/trending
Hit 2000 Stars in 7 days https://github.com/yujiosaka/headless-chrome-crawler > 2000
ເ͕ͬͨ
ظ͍ͯͨ͠ϝϦοτ ✓ ࣭ͷ্ ✓ ։ൃྗͷ্ ✓ ϒϥϯυ্ ✓ ࣄʹͭͳ͕Δ ୭͔ࣄ͘ΕΖ͍ͩ͘͞
ҙ֎ͱࣗݾΞϐʔϧஏ͔͍ͣ͠ ·ͩλΠϙमਖ਼͔͠ૹΒΕͯͳ͍ IssueΒϝʔϧΒରԠ͍͠
ࣄதͣͬͱχϠχϠͰ͖Δ
ຊ ଟ͘ͷਓͷʹཹ·ͬͨϥΠϒϥϦͱ ͦ͏ͳΒͳ͔ͬͨϥΠϒϥϦͷҧ͍Λཧ ※ͨͬͨαϯϓϧ̍ͷ͜ͱͳͷͰʹฉ͍͍ͯͩ͘͞
λʔήοτͷ͞ ✓ ӳޠͰൃ৴͢Δॏཁੑ ✓ READMEΛӳޠͰॻ͍ͨΒ͓ऴ͍Ͱͳ͍ ✓ ʰͯϒϗοτΤϯτϦʔʱΑΓ ʰHacker News Top
Linksʱͷํ͕10ഒྲྀೖ͕͋ͬͨ
ར༻ͷϋʔυϧ ✓ ҰݟͯԿ͕͍ͨ͠ͷ͔ϋοΩϦ͔Δ ✓ Headless Chrome + Crawler = Headless
Chrome Crawler ✓ READMEͰϝϦοτΛҰ൪࠷ॳʹΞϐʔϧ͓ͯ͘͠ ✓ ͦΕͰ͔Βͳ͍ਓ͚ʹɺFAQΛ༻ҙ͢Δ ✓ ͍͍ͩͨͷਓ࠷ॳͷߦ͔͠ಡ·ͳ͍ ✓ ը૾ϩΰͰʮ؆୯͞ʯΞϐʔϧͰ͖Δ
ίʔυϦʔσΟϯάͷϋʔυϧ ✓ examples Λॆ࣮ͤ͞Δ ✓ Ұ൪ಡ·Ε͍ͯͨίϯςϯπ examples ͩͬͨ ✓ πʔϧϑϧ׆༻͢Δ
✓ ESLint ✓ commitlint ✓ EditorConfig ✓ TypeScript / JSDoc support
৴པ ✓ όοδΛϑϧ׆༻ ✓ ࠷৽ͷϏϧυঢ়گ͕ҰͰ͔ΔΑ͏ʹ͢Δ ✓ άϦʔϯ͕ฒΜͰΔͱͳΜ͔҆৺͢Δ ✓ Greenkeeper࠷ڧઆ ✓
࠷ޙͷίϛοτ͕લͷϓϩδΣΫτͱ͔͍ͨ͘ͳ͍ ✓ ͠Β͘αϘͬͯͯɺৗʹίϛοτͰ͖Δ
Happy Niya-niya Hacking!
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
370
Machine Learning with JavaScript
yujiosaka
0
250
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
510
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.5k
『XXX』のための管理画面
yujiosaka
1
1.5k
Enjoy Deep Learning by JavaScript
yujiosaka
1
420
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
630
第二回もんご祭 パネルディスカッション
yujiosaka
0
960
Other Decks in Technology
See All in Technology
個人ASを取得して、自宅Kubernetes上のVMをグローバルIPで公開した話
munenick
0
210
閉域網でのAPI公開の選択肢全部比較!
takano0131
4
560
お風呂場をサーバルームにしよう
yusuke427
0
360
小さなエージェントから始めてみよう!実践コンテキストエンジニアリング
iwamot
PRO
0
130
飯から始まるデータエンジニア
sansantech
PRO
0
110
専門家に使ってもらうAIプロダクトをつくる - 病院の医事課向けDPCコーディング支援システムの設計と評価
hagino3000
4
850
re:Inventを楽しむために、 1日1つ「これが楽しみ」を決めよう
beli68
0
120
IoTのデータ保存をSREの観点で考える
grimoh
1
110
LLM 基盤と kagent v0 を全部 k8s に入れてみた / Deploy both LLM and kagent v0 in k8s
kota2and3kan
0
240
gem install とrequire のむこうがわ
ydah
0
220
変更差分を読むために環境を整備しよう
pipepasta
1
470
AWS Emulator を評価する: 手元で動く AWS の API の仕組み、始め方、確かめ方
inariku
PRO
2
760
Featured
See All Featured
SEO in 2025: How to Prepare for the Future of Search
ipullrank
3
3.9k
Amusing Abliteration
ianozsvald
1
390
Introduction to Domain-Driven Design and Collaborative software design
baasie
1
1.1k
Public Speaking Without Barfing On Your Shoes - THAT 2023
reverentgeek
1
670
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
23k
A Guide to Academic Writing Using Generative AI - A Workshop
ks91
PRO
1
570
Evolving SEO for Evolving Search Engines
ryanjones
0
360
[RailsConf 2023] Rails as a piece of cake
palkan
59
7.2k
Facilitating Awesome Meetings
lara
57
7.2k
The Limits of Empathy - UXLibs8
cassininazir
2
790
More Than Pixels: Becoming A User Experience Designer
marktimemedia
3
630
AI Search: Where Are We & What Can We Do About It?
aleyda
0
8.2k