Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
AIと私たちの学習の変化を考える - Claude Codeの学習モードを例に
Avatar for azukiazusa azukiazusa1
10
3.9k
Laravel Boost 超入門
Avatar for Arlo fire_arlo
3
210
@Environment(\.keyPath)那么好我不允许你们不知道！ / atEnvironment keyPath is so good and you should know it!
Avatar for Elvis Shi lovee
0
120
FindyにおけるTakumi活用と脆弱性管理のこれから
Avatar for adachi.ryo rvirus0817
0
500
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
Avatar for Iury Souza iurysza
0
140
Go言語での実装を通して学ぶLLMファインチューニングの仕組み / fukuokago22-llm-peft
Avatar for monochromegane monochromegane
0
120
今から始めるClaude Code入門〜AIコーディングエージェントの歴史と導入〜
Avatar for nokomoro3 nokomoro3
0
130
さようなら Date。 ようこそTemporal！ 3年間先行利用して得られた知見の共有
Avatar for 8-beeeaaat!!! 8beeeaaat
3
1.4k
モバイルアプリからWebへの横展開を加速した話_Claude_Code_実践術.pdf
Avatar for Kazuya Sakamoto kazuyasakamoto
0
320
Ruby Parser progress report 2025
Avatar for yui-knk yui_knk
1
440
Kiroの仕様駆動開発から見えてきたAIコーディングとの正しい付き合い方
Avatar for clshinji clshinji
1
210
How Android Uses Data Structures Behind The Scenes
Avatar for HyunWoo Lee l2hyunwoo
0
440

Featured

See All Featured
Navigating Team Friction
Avatar for Lara Hogan lara
189
15k
How GitHub (no longer) Works
Avatar for Zach Holman holman
315
140k
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
70
11k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
53k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
57
5.8k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
29
5.5k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
7
840
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
31
2.2k
Making Projects Easy
Avatar for Brett Harned brettharned
117
6.4k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
3
620

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412