Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Scrapy Overview

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
210
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
クラウドエージェントは、 何がそんなに便利なのか / what makes cloud agents so useful
Avatar for r-kagaya rkaga
3
600
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか？〜 / Introduction to IaC in the AI Era
Avatar for アキキー | Akihisa Ikeda akihisaikeda
5
1.1k
全ての永続トークンを、生まれる前に消し去りたい！
Avatar for sylph01 sylph01
2
1.5k
3回つくってわかった、
定期実行ジョブ基盤の本質情報
Avatar for Naoto Takai takai
4
3.3k
graphql-ruby を Spinel で速くしようとした話
Avatar for Fumiaki MATSUSHIMA mtsmfm
0
120
Snowflake Openflow Connector利用の勘所 / 20261024developersio2026tokyo-snowflake-openflow
Avatar for kasacchiful kasacchiful
PRO
0
220
自サークルのIdPからログアウトできるようになるまで
Avatar for taiseiue taiseiue
1
150
現場と育てるRailsモデリング 〜Spreadsheetから始めたモデル設計〜
Avatar for siroemk siroemk
2
1.1k
JIS keyboard support in Omarchy
Avatar for LEF lef237
0
160
contenteditable と日本語入力に向き合う
Avatar for taki colorful12
0
230
Apple Intelligenceとオンデバイス時代の到来
Avatar for Yahiro Sugiyama yahir0
0
160
Groovy as a human and AI friendly programming language
Avatar for paulking paulk
0
120

Featured

See All Featured
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
1.1k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
96
6.9k
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
163
24k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
76
5.4k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
600
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
550
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
371
20k
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
430
Claude Code のすすめ
Avatar for schroneko schroneko
67
230k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
Chasing Engaging Ingredients in Design
Avatar for Sebastian Deterding codingconduct
0
370
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
143
7.8k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412