Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky
Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
DMMを支える決済基盤の技術的負債にどう立ち向かうか / Addressing Technical Debt in Payment Infrastructure
Avatar for yoshiyoshifujii yoshiyoshifujii
3
220
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
Avatar for hideki kinjyo o0h
PRO
1
340
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
Avatar for shiro seike seike460
PRO
3
790
LT 2025-06-30: プロダクトエンジニアの役割
Avatar for Keisuke Yamamoto yamamotok
0
850
NPOでのDevinの活用
Avatar for みんなのコード codeforeveryone
0
890
AI時代の『改訂新版 良いコード／悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
Avatar for MinoDriven minodriven
23
9.4k
ソフトウェア品質を数字で捉える技術。事業成長を支えるシステム品質の マネジメント
Avatar for takuya542 takuya542
2
15k
チームのテスト力を総合的に鍛えて品質、スピード、レジリエンスを共立させる/Testing approach that improves quality, speed, and resilience
Avatar for Hiroki Iseri goyoki
5
1.1k
ISUCON研修おかわり会 講義スライド
Avatar for Yuki Yata arfes0e2b3c
1
460
ふつうの技術スタックでアート作品を作ってみる
Avatar for Akira Yagi akira888
1
1.3k
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
Avatar for shimabox shimabox
2
610
20250708_JAWS_opscdk
Avatar for Takuya Yonezawa takuyay0ne
2
130

Featured

See All Featured
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
71
11k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.7k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
77
5.9k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
90
590k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.7k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
16k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
970
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
54
11k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.7k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
31
1.3k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412