Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky
Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
iOSアプリで測る！名古屋駅までの 方向と距離
Avatar for Ryu-nakayama ryunakayama
0
160
KawaiiLT 登壇資料　キャリアとモチベーション
Avatar for 柊 hiiragi
0
160
In geheimer Mission: AI Agents entwickeln
Avatar for Jörg Neumann joergneumann
0
110
Cursorを活用したAIプログラミングについて 入門
Avatar for 株式会社レクト rect
0
190
generative-ai-use-cases(GenU)の推しポイント ～2025年4月版～
Avatar for Hideyuki Nagata hideg
1
390
eBPF超入門「o11yに使える」とは (20250424_eBPF_o11y)
Avatar for Koki Senda thousanda
1
120
「理解」を重視したAI活用開発
Avatar for FastDOCTOR fast_doctor
0
300
REALITY コマンド作成チュートリアル
Avatar for にしうり らく nishiuriraku
0
120
2025年のz-index設計を考える
Avatar for TAK tak_dcxi
11
4k
KANNA Android の技術的課題と取り組み
Avatar for Yosuke Watanabe watabee
1
460
はじめてのPDFKit.pdf
Avatar for shoma.kato shomakato
0
100
The New Developer Workflow: How AI Transforms Ideas into Code
Avatar for Daniel Sogl danielsogl
0
120

Featured

See All Featured
Building Adaptive Systems
Avatar for Chris Keathley keathley
41
2.5k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
7
420
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
41
2.6k
Building an army of robots
Avatar for Kyle Neath kneath
305
45k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
6
540
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
267
13k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
31
1.2k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.1k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
205
24k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
16k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
160
15k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412