Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
コントリビューターによるDenoのすゝめ / Deno Recommendations by a Contributor
Avatar for petamoriken / 森建 petamoriken
0
200
Honoを使ったリモートMCPサーバでAIツールとの連携を加速させる！
Avatar for とすり tosuri13
1
170
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
650
Fluid Templating in TYPO3 14
Avatar for Simon Praetorius s2b
0
120
humanlayerのブログから学ぶ、良いCLAUDE.mdの書き方
Avatar for Yuto tsukamoto1783
0
170
Apache Iceberg V3 and migration to V3
Avatar for Tomohiro Tanaka tomtanaka
0
130
副作用をどこに置くか問題：オブジェクト指向で整理する設計判断ツリー
Avatar for Koya Masuda koxya
1
570
Data-Centric Kaggle
Avatar for ねぼすけAI isax1015
2
750
AI 駆動開発ライフサイクル(AI-DLC):ソフトウェアエンジニアリングの再構築 / AI-DLC Introduction
Avatar for kanamsasa kanamasa
11
6.2k
KIKI_MBSD Cybersecurity Challenges 2025
Avatar for ikema ikema
0
1.3k
AI時代の認知負荷との向き合い方
Avatar for OptFit Corp. optfit
0
130
2026年 エンジニアリング自己学習法
Avatar for yumechi(Motoki Hirao) yumechi
0
120

Featured

See All Featured
Ethics towards AI in product and experience design
Avatar for Skipper Chong Warson skipperchong
2
190
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.7k
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
1
230
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
110
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
5.2k
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
280
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.3k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
44
Test your architecture with Archunit
Avatar for Yoan thirion
1
2.1k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412