Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
ここ半年くらいでAIに作らせたR用ツール
Avatar for eitsupi eitsupi
0
340
信頼性について考えてみる（SRE NEXT 2026 miniLT）
Avatar for hayama hayama17
0
230
Claude Team Plan導入・ガイド
Avatar for tk3fftk tk3fftk
0
240
Apache Hive: そしてCloud Native Lakehouseへ
Avatar for okumin okumin
1
190
そこに3びきプロダクトがいるじゃろう——生成AI時代における“価値が届かない理由”の構造
Avatar for kosuke T kosuket
0
280
Laravelで学ぶ Webアプリケーションチューニング入門/web_application_tuning_101
Avatar for Ryo Tomidokoro hanhan1978
4
1.4k
torikago - Ruby::Boxで照らすモジュラモノリスの実行境界
Avatar for nori se4weed
1
260
PostgreSQL 18で考えるUUID主キー
Avatar for Kazuhiro Seo kazuhiro1982
0
440
地域 SRE コミュニティ最前線 - ホンマでっかSRE勉強会
Avatar for tk3fftk tk3fftk
0
290
jsmini JavaScript Engine を作ってみた話
Avatar for Yosuke Furukawa yosuke_furukawa
PRO
0
260
Embedded SREと共に達成した会員管理システムのAWS移行 - SRE NEXT 2026 ランチスポンサーセッション
Avatar for ニフティ株式会社 niftycorp
PRO
1
3.2k
「正の参照」と 「負の導出」で組む ハーネスエンジニアリング
Avatar for Masahiro Watanabe cottpan
1
150

Featured

See All Featured
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.7k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6k
Ruling the World: When Life Gets Gamed
Avatar for Sebastian Deterding codingconduct
0
290
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
470
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
640
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
270
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
340
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Lightning Talk: Beautiful Slides for Beginners
Avatar for Ines Montani inesmontani
PRO
2
610
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412