Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
高専、大学編入、そして未踏へ〜プロダクト開発とキャリアの歩み - Technical College, University Transfer, and On to “Mitou” / My Journey in Product Development and Career
Avatar for __pkmiya pkmiya
0
140
信頼性の目標を誰も求めてない
Avatar for Shuma shubox
0
490
新人はどこまで自力でやり、どこからAIに頼るべきか／エンジニア育成に向き合う_先輩たちの悩みと知見共有会
Avatar for TOPPAN ict toppan_digital_dev
1
560
まずはプロンプトガイドを読もう、話はそれからだ
Avatar for Akira TSURUDA kiakiraki
1
260
XP祭りでしか伝わらないフリップネタ #xpjug
Avatar for bayashi murabayashi
0
110
AIを上手に使っていこうとしたら越境せざるを得なくなった話 〜実践1年で見えた境界を越えなければならない理由と進め方〜 / Crossing borders with AI
Avatar for TomoyaKitaura tomoyakitaura
3
1k
片田舎のおっさん、 Swift Buildのダイアモンド問題解決の不具合修正PRを出すが、解決方法がキャッシュをしないようにすることであり、ビルド時間が伸びると言われてマージされないので高速化もする/swiftbuild
Avatar for yimajo yimajo
0
360
KotlinConf Extended South Korea 2026 Keynote
Avatar for HyunWoo Lee l2hyunwoo
0
130
Foundry Localでエージェント開発
Avatar for 瀬尾佳隆 seosoft
0
130
AIは賢い。でも実行環境は？ CLIおじさんがAI時代に伝えたいこと ~ CLIおじさんがAI時代に伝えたいこと ~
Avatar for curekoshimizu curekoshimizu
1
150
DroidKaigi 2026 「個人開発という実験場: Android エンジニアが手にする4つの自由」
Avatar for SlashNephy slashnephy
0
220
「AI時代、配布するPythonコードをどう守るか: 難読化の実験と判断軸」 #PyconJP2026
Avatar for PKSHA Technology（パークシャテクノロジー） pkshadeck
PRO
2
160

Featured

See All Featured
Sam Torres - BigQuery for SEOs
Avatar for Tech SEO Connect techseoconnect
PRO
0
530
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
400
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
380
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.5k
Deep Space Network (abreviated)
Avatar for Tony Rice tonyrice
0
290
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.5k
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
920
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
430
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
3
230
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.3k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412