Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
200

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
What Spring Developers Should Know About Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
440
Claude Code の Skill で複雑な既存仕様をすっきり整理しよう
Avatar for yuichiro_takahashi yuichirokato
1
370
「やめとこ」がなくなった — 1月にZennを始めて22本書いた AI共創開発のリアル
Avatar for atani atani14
0
370
API Platformを活用したPHPによる本格的なWeb API開発 / api-platform-book-intro
Avatar for Takashi Kanemoto ttskch
1
130
ロボットのための工場に灯りは要らない
Avatar for watany watany
10
2.7k
Docコメントで始める簡単ガードレール
Avatar for Keisuke Ikeda keisukeikeda
1
110
「抽象に依存せよ」が分からなかった新卒1年目の私が Goのインターフェースと和解するまで
Avatar for Genki-Miyakawa kurogenki
0
110
エンジニアの「手元の自動化」を加速するn8n 2026.02.27
Avatar for 西尾義英 symy2co
0
140
Agent Skills Workshop - AIへの頼み方を仕組み化する
Avatar for Gota gotalab555
15
8.6k
SourceGeneratorのマーカー属性問題について
Avatar for tkym htkym
0
190
Railsの気持ちを考えながらコントローラとビューを整頓する/tidying-rails-controllers-and-views-as-rails-think
Avatar for MOROHASHI Kyosuke moro
5
390
maplibre-gl-layers - 地図に移動体たくさん表示したい
Avatar for Kouji Matsui kekyo
PRO
0
260

Featured

See All Featured
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
390
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8k
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
9
36k
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
Avatar for Katarina Dahlin katarinadahlin
PRO
1
3.5k
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
180
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
72
12k
Highjacked: Video Game Concept Design
Avatar for Ryan Kendrick rkendrick25
PRO
1
310
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
55
12k
HDC tutorial
Avatar for Michiel Stock michielstock
1
530
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
480
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412