Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
読もう! Android build ドキュメント
andpad
1
240
php-fpm がリクエスト処理する仕組みを追う / Tracing-How-php-fpm-Handles-Requests
shin1x1
5
830
Going Structural with Named Tuples
bishabosha
0
170
令和トラベルにおけるコンテンツ生成AIアプリケーション開発の実践
ippo012
1
260
マルチアカウント環境での、そこまでがんばらない RI/SP 運用設計
wa6sn
0
600
PHPer's Guide to Daemon Crafting Taming and Summoning
uzulla
2
1.1k
Preact、HooksとSignalsの両立 / Preact: Harmonizing Hooks and Signals
ssssota
1
710
Coding Experience Cpp vs Csharp - meetup app osaka@9
harukasao
0
120
本当だってば！俺もTRICK 2022に入賞してたんだってば！
jinroq
0
250
複雑なフォームと複雑な状態管理にどう向き合うか / #newt_techtalk vol. 15
izumin5210
4
3.3k
S3静的ホスティング+Next.js静的エクスポート で格安webアプリ構築
iharuoru
0
200
いまさら聞けない生成AI入門: 「生成AIを高速キャッチアップ」
soh9834
12
3.8k

Featured

See All Featured
Why Our Code Smells
bkeepers
PRO
336
57k
Building Adaptive Systems
keathley
41
2.5k
Building Flexible Design Systems
yeseniaperezcruz
328
38k
GraphQLとの向き合い方2022年版
quramy
45
14k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
4
470
Building an army of robots
kneath
304
45k
A Modern Web Designer's Workflow
chriscoyier
693
190k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
12
1.4k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
21k
Building Applications with DynamoDB
mza
94
6.3k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
Mobile First: as difficult as doing things right
swwweet
223
9.5k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412