Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
Pythonでもちょっとリッチな見た目のアプリを設計してみる
ueponx
1
570
GoとPHPのインターフェイスの違い
shimabox
2
190
1年目の私に伝えたい！テストコードを怖がらなくなるためのヒント/Tips for not being afraid of test code
push_gawa
0
190
CDK開発におけるコーディング規約の運用
yamanashi_ren01
2
130
JavaScriptツール群「UnJS」を5分で一気に駆け巡る！
k1tikurisu
9
1.8k
Grafana Loki によるサーバログのコスト削減
mot_techtalk
1
130
Linux && Docker 研修/Linux && Docker training
forrep
24
4.5k
Honoをフロントエンドで使う 3つのやり方
yusukebe
7
3.3k
『GO』アプリ バックエンドサーバのコスト削減
mot_techtalk
0
140
CSS Linter による Baseline サポートの仕組み
ryo_manba
1
110
データの整合性を保つ非同期処理アーキテクチャパターン / Async Architecture Patterns
mokuo
47
17k
color-scheme: light dark; を完全に理解する
uhyo
5
380

Featured

See All Featured
The Cult of Friendly URLs
andyhume
78
6.2k
The Cost Of JavaScript in 2023
addyosmani
47
7.3k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
12
960
Typedesign – Prime Four
hannesfritz
40
2.5k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
Practical Orchestrator
shlominoach
186
10k
How to Ace a Technical Interview
jacobian
276
23k
Bootstrapping a Software Product
garrettdimon
PRO
306
110k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Done Done
chrislema
182
16k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412