Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
並行開発のためのコードレビュー
Avatar for Miyuki miyukiw
2
1.9k
2026年は Rust 置き換えが流行る！ / 20260220-niigata-5min-tech
Avatar for girigiribauer girigiribauer
0
180
今、アーキテクトとして 品質保証にどう関わるか
Avatar for Nealle nealle
0
170
Raku Raku Notion 20260128
Avatar for hareyaka hareyakayuruyaka
0
410
CSC307 Lecture 10
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
690
AIによる高速開発をどう制御するか？ ガードレール設置で開発速度と品質を両立させたチームの事例
Avatar for tonkotsuboy_com tonkotsuboy_com
7
2.5k
AIエージェントのキホンから学ぶ「エージェンティックコーディング」実践入門
Avatar for Masahiro Nishimi masahiro_nishimi
7
1.1k
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
170
Oxlintはいいぞ
Avatar for Yuji Yamaguchi yug1224
5
1.5k
AIとペアプロして処理時間を97%削減した話 #pyconshizu
Avatar for Kashun Yoshida kashewnuts
1
110
SourceGeneratorのススメ
Avatar for tkym htkym
0
620
PostgreSQL を使った快適な go test 環境を求めて
Avatar for Kotaro Otaka otakakot
0
120

Featured

See All Featured
Leo the Paperboy
Avatar for Maya Tellez mayatellez
4
1.4k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
9.9k
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
2
270
How to build a perfect <img>
Avatar for Jono Alderson jonoalderson
1
5.2k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.2k
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
62
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.7k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
680
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412