Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Scrapy Overview
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
JusBrasil
April 12, 2013
Programming
210
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Scrapy Overview
An overview of the Scrapy framework by @cacovsky
JusBrasil
April 12, 2013
Transcript
50 slides
Scrapy an overview
/skræpi/
Web Crawler vs. Web Scraper
None
None
Scrapy Framework Scraping / Crawling / Monitoring / Testing
Stable Active Large community
~200 pages of docs
Commercial support
Framework?
None
None
None
Twisted event loop (reactor)
None
Your code goes here
The scraping logic
None
HttpErrorMiddleware UrlLengthMiddleware DepthMiddleware
HttpProxyMiddleware HttpCacheMiddleware RedirectMiddleware
Media download Persistence Post-processing
Data flow control
Queuing
Talk is cheap, show me the code.
$ pip install Scrapy $ scrapy startproject home_news
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Project root
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Project config
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Project module
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Your items
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Your pipelines
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Your settings
home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
Your spiders...
None
//*[@id="glbcorpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href
//*[@id="glbmateria"]/div[2]/h1/text()
//*[@id="materialetra"]/div/div/p[1]/text()
None
$ pwd /home/caco/studies/scrapy_news/home_news
$ pwd /home/caco/studies/scrapy_news/home_news (project root)
$ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t
json
$ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t
json
$ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t
json
$ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t
json (feed exporters: json,csv,xml)
None
None
None
Other nice features • scrapyd: run as a service •
Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)
Obrigado! @cacovsky Thanks! @cacovsky
Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412
Other Decks in Programming
See All in Programming
ビジネスロジックを堅牢に~Scala 3のドメインプリミティブ~
j5ik2o
0
210
私なりのハーネスエンジニアリング ~独自の言語処理系の開発を通して~
seosoft
0
230
Crash Auto Pipelines
veronikapj
1
440
Apple Intelligenceとオンデバイス時代の到来
yahir0
0
250
AppFunctionsの内部実装を紐解く
hiroaki404
0
100
[Kotlin Fest 2026] eBPF Tracing for Kotlin
takehaya
1
390
手の動かし方がわからなかった私が個人開発者になるまで
usagi0920_pochi
0
320
Scalaの型でマルチテナントの認可を守る
nextbeatdev
0
160
Ruby から Emacs を操作したい
phigasui
0
370
Zero-Tap Sign-In対応、結局何をすればいいの?
mitohato14
0
2.6k
Snowflake Openflow Connector利用の勘所 / 20261024developersio2026tokyo-snowflake-openflow
kasacchiful
PRO
0
360
Claude Code ハーネスDeep Dive – 君はどこまで制御する? -
takateeen_knj
6
6.2k
Featured
See All Featured
How to Ace a Technical Interview
jacobian
280
24k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Game over? The fight for quality and originality in the time of robots
wayneb77
1
340
Visualization
eitanlees
153
17k
Designing for Performance
lara
612
70k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
rkaga
63
48k
Why Our Code Smells
bkeepers
PRO
340
58k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
6.2k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
46
3.1k
Thoughts on Productivity
jonyablonski
76
5.5k
Measuring Dark Social's Impact On Conversion and Attribution
stephenakadiri
2
330
Paper Plane
katiecoart
PRO
6
55k