Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
AIレビュアーをスケールさせるには / Scaling AI Reviewers
Avatar for technuma technuma
2
210
WebAssemblyインタプリタを書く ～Component Modelを添えて～
Avatar for ruccho ruccho
1
870
Scale out your Claude Code ~⁨⁩自社専用Agentで10xする開発プロセス~
Avatar for Yuku Kotani yukukotani
9
2.4k
DockerからECSへ 〜 AWSの海に出る前に知っておきたいこと 〜
Avatar for Itaru Ota ota1022
5
1.7k
オープンセミナー2025@広島 「君はどこで動かすか？」 アンケート結果
Avatar for Satoshi Kaneyasu satoshi256kbyte
0
150
Portapad紹介プレゼンテーション
Avatar for KAKERU gotoumakakeru
1
130
Flutterと Vibe Coding で個人開発！
Avatar for Hayami Shuhei hyshu
1
260
学習を成果に繋げるための個人開発の考え方 〜 「学習のための個人開発」のすすめ / personal project for leaning
Avatar for panda_program panda_program
1
100
Vibe coding コードレビュー
Avatar for kinopee kinopeee
0
460
兎に角、コードレビュー
Avatar for mitohato14 mitohato14
0
140
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
Avatar for はる lemonade_37
2
480
Webinar: AI-Powered Development: Transformiere deinen Workflow mit Coding Tools und MCP Servern
Avatar for Daniel Sogl danielsogl
0
140

Featured

See All Featured
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
44
2.4k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.2k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
16k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
338
57k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
33
8.8k
Scaling GitHub
Avatar for Zach Holman holman
462
140k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
1k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
283
13k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
46
7.6k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
51k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
71
11k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412