Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Vite+ Unified Toolchain for the Web
Avatar for Naoki Haba naokihaba
0
410
1B+ /day規模のログを管理する技術
Avatar for Broadleaf Co., Ltd. broadleaf
0
120
Go1.27で導入されるジェネリクスメソッドでできること
Avatar for mackee mackee
0
230
Oxcを導入して開発体験が向上した話
Avatar for Yuji Yamaguchi yug1224
4
370
Semantic Version 単位で戦略を柔軟に変えて、パッケージアップデートを自動化する
Avatar for daitasu daitasu
1
330
AIで効率化できた業務・日常
Avatar for Ochtum ochtum
0
160
OS アップデート対応の取り組み方がもっと共有されてほしい
Avatar for ANDPAD inc andpad
0
110
Agentic UI
Avatar for Manfred Steyer manfredsteyer
PRO
0
220
吝嗇家のためのAI活用 / AI development for miser - ChatGPT + Issue Driven Development
Avatar for philomagi tooppoo
0
160
作って学ぶ、 JSX (TSX) ランタイムの基本
Avatar for syumai syumai
7
1.7k
コンテキストの使い捨てをやめる — ビジネスルール駆動開発と miko —
Avatar for ioki ioki
0
260
act1-costs.pdf
Avatar for Sumedh Bala sumedhbala
0
150

Featured

See All Featured
Heart Work Chapter 1 - Part 1
Avatar for Lake Fama lfama
PRO
8
36k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
191
11k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.2k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8.2k
A Soul's Torment
Avatar for Nat Ma seathinner
6
3k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
230
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
34k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
96
14k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
76
5.2k
技術選定の審美眼（2025年版） / Understanding the Spiral of Technologies 2025 edition
Avatar for Takuto Wada twada
PRO
118
120k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412