Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
200

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
Go Conference mini in Sendai 2026 : Goに新機能を提案し実装されるまでのフロー徹底解説
Avatar for Takahito Yamatoya yamatoya
0
570
AI時代のシステム設計：ドメインモデルで変更しやすさを守る設計戦略
Avatar for 増田 亨 masuda220
PRO
5
920
How to stabilize UI tests using XCTest
Avatar for Akio Itaya akkeylab
0
120
Agent Skills Workshop - AIへの頼み方を仕組み化する
Avatar for Gota gotalab555
15
8.6k
コードレビューをしない選択 #でぃーぷらすトウキョウ
Avatar for Takuma Kajikawa kajitack
3
920
Windows on Ryzen and I
Avatar for 瀬尾佳隆 seosoft
0
260
野球解説AI Agentを開発してみた - 2026/02/27 LayerX社内LT会資料
Avatar for Shinichi Nakagawa shinyorke
PRO
0
260
encoding/json/v2のUnmarshalはこう変わった：内部実装で見る設計改善
Avatar for Hiroki Kurasawa kurakura0916
0
400
Docコメントで始める簡単ガードレール
Avatar for Keisuke Ikeda keisukeikeda
1
110
ポーリング処理廃止によるイベント駆動アーキテクチャへの移行
Avatar for Seitaro Fujigaki seitarof
3
1k
Ruby x Terminal
Avatar for Akira Matsuda a_matsuda
7
590
メタプログラミングで実現する「コードを仕様にする」仕組み/nikkei-tech-talk43
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
180

Featured

See All Featured
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
300
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
100
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
480
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.8k
What the history of the web can teach us about the future of AI
Avatar for Ines Montani inesmontani
PRO
1
470
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.6k
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
273
27k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
5
940
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
75
5.1k
職位にかかわらず全員がリーダーシップを発揮するチーム作り / Building a team where everyone can demonstrate leadership regardless of position
Avatar for MADOX madoxten
61
52k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
2.6M

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412