Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Running Swift without an OS
Avatar for Kishikawa Katsumi kishikawakatsumi
0
750
Oxlintとeslint-plugin-react-hooks 明日から始められそう？
Avatar for t6adev t6adev
0
120
Mastering Event Sourcing: Your Parents Holidayed in Yugoslavia
Avatar for Marek Matulka super_marek
0
150
L’IA au service des devs : Anatomie d'un assistant de Code Review
Avatar for Thomas Boileau toham
0
220
The Monolith Strikes Back: Why AI Agents ❤️ Rails Monoliths
Avatar for Rodrigo Serradura serradura
0
300
今年もTECHSCOREブログを書き続けます！
Avatar for 平奥真一（ひらおくしんいち） hiraoku101
0
240
PHPのバージョンアップ時にも役立ったAST（2026年版）
Avatar for Atsushi Matsuo matsuo_atsushi
0
300
年間50登壇、単著出版、雑誌寄稿、Podcast出演、YouTube、CM、カンファレンス主催……全部やってみたので面白さ等を比較してみよう / I’ve tried them all, so let’s compare how interesting they are.
Avatar for nrs nrslib
4
770
Java 21/25 Virtual Threads 소개
Avatar for Sunghyouk Bae (Debop) debop
0
340
ルールルルルルRubyの中身の予備知識 ── RubyKaigiの前に予習しなイカ？
Avatar for ydah ydah
0
120
forteeの改修から振り返るPHPerKaigi 2026
Avatar for muno92 muno92
PRO
3
260
ドメインイベントでビジネスロジックを解きほぐす #phpcon_odawara
Avatar for Takuma Kajikawa kajitack
2
130

Featured

See All Featured
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
160
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
5.6k
New Earth Scene 8
Avatar for Sydney Stone popppiees
3
2k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
40
2.3k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.1k
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
310
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2k
Jess Joyce - The Pitfalls of Following Frameworks
Avatar for Tech SEO Connect techseoconnect
PRO
1
130
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
Avatar for Aleyda Solis aleyda
1
1.9k
Efficient Content Optimization with Google Search Console & Apps Script
Avatar for Katarina Dahlin katarinadahlin
PRO
1
490

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412