Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
How to scrape web contents in Clojure
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
ayato
January 09, 2016
Programming
120
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
How to scrape web contents in Clojure
ayato
January 09, 2016
Transcript
9 slides
)PXUPTDSBQF XFCDPOUFOUT JO$MPKVSF !@BZBUP@Q
͋ͽʔ $MPKVSJBO $ZCP[V4UBSUVQT *OD
8IBUJTXFCTDSBQJOH ΣϒεΫϨΠϐϯά 8FCTDSBQJOH ͱɺ ΣϒαΠτ͔ΒใΛநग़͢Δ ίϯϐϡʔλιϑτΣΞٕज़ͷ͜ͱɻ CZXJLJQFEJB
1SPCMFNT 8FCίϯςϯπߏʹ͍ۙܗΛ͍ͯ͠Δ ࣅ͍ͯΔϖʔδ͕ࢁ͋Δ͕ඍົʹҧ͏ ߏΛ୧Δ࠶ؼతͳίʔυΛॻ͘ඞཁ͕͋Δ ͍͍ͩͨ໘͍͘͞
4LZTDSBQFS ߏΛ࠶ؼతʹ୧ͬͯ͘ΕΔ ϖʔδͷλΠϓຖʹॲཧํ๏͚ͩॻ͚͍͍ ԆγʔέϯεΛฦͯ͘͠ΕΔ Ωϟογϡػߏ͕͍͍ͭͯΔ εΫϨΠϐϯά෦&OMJWFґଘ IUUQTHJUIVCDPNOBUIFMMTLZTDSBQFS
(defn seed [username from until] (let [url (str "http://twilog.org/" username)]
[{:username username :from from :until until :url url :processor ::user-page}])) (s/defprocessor user-page :cache-template "twilog/:username" :process-fn (fn [res {:keys [username]}] (let [not-registered (seq (html/select res [:div.box-info.box-icon])) not-found (seq (html/select res [:div.box-attention.box-icon]))] (cond not-registered [{:msg "This account was not registered."}] not-found [{:msg "This account was not found."}] :else [{:url (str "http://twilog.org/" username "/archives") :processor ::archives-page}])))) &YBNQMF
(defn scrape [username & [{:as options :keys [html-cache processed-cache from
until] :or {html-cache true processed-cache true from "00000000" until "99999999"}}]] (let [handler (create-handler identity options)] (handler (s/scrape (seed username from until) :html-cache html-cache :processed-cache processed-cache)))) &YBNQMF
$PODMVTJPO 4LZTDSBQFSΛ͏ͱͤʹͳΕΔ $MPKVSF࠷ߴʂ
Enjoy Clojure
More Decks by ayato
See All by ayato
マイクロサービス内で動くAPIをF#で書いている
ayato0211
1
1.8k
Clojureという言語が私逹にもたらしたもの
ayato0211
6
3.2k
3年間考え続けてきたWebアプリケーションにおけるテストの話
ayato0211
3
330
Re:REPL-Driven Development
ayato0211
3
1.4k
Meta Template Engine
ayato0211
2
1.2k
超変換! Hiccup data structure!!
ayato0211
2
700
About Integrant
ayato0211
0
620
Muscle Assert
ayato0211
0
340
Clojureを用いたWebアプリケーション開発
ayato0211
2
3.3k
Other Decks in Programming
See All in Programming
コード生成時代のAPI定義を支えるRequest Spec移行
pvcresin
0
280
私なりのハーネスエンジニアリング ~独自の言語処理系の開発を通して~
seosoft
0
220
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
leegeunhyeok
0
180
AI-Powered Domain-Driven Design
ziobrando
0
470
flutter_test のその先へ — Flutter アプリの LLM 機能に「評価」という品質保証を
kosukesaigusa
0
200
GraphQL の onError と Nullability のこれから
quramy
0
220
Kotlin JVM用に作られたライブラリのKMPを前提とした再設計
jsoizo
0
200
SODA - FACT BOOK(JP)
sodainc
1
40k
Kawaiicon: Kubernetes, it's no longer cool, but let's secure it anyway!
barnbarn
0
140
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
2
190
ローカルLLMの基本と近況 2611 / Local LLM basics and recent situation 2611
kishida
3
590
AI時代のTerminal環境を UNIX哲学で設計・運用する
yasunori0418
1
360
Featured
See All Featured
The Spectacular Lies of Maps
axbom
PRO
1
1.1k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
inesmontani
PRO
3
2.5k
Paper Plane (Part 1)
katiecoart
PRO
3
12k
Mozcon NYC 2025: Stop Losing SEO Traffic
samtorres
1
600
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
47
8.4k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
akihiro_kokubo
PRO
74
43k
The B2B funnel & how to create a winning content strategy
katarinadahlin
PRO
1
620
Un-Boring Meetings
codingconduct
0
500
The #1 spot is gone: here's how to win anyway
tamaranovitovic
4
1.2k
From Legacy to Launchpad: Building Startup-Ready Communities
dugsong
0
380
Visual Storytelling: How to be a Superhuman Communicator
reverentgeek
2
750
Music & Morning Musume
bryan
48
7.5k