Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Casual Log Collection and Querying with fluent-...
Search
UENISHI Kota
June 01, 2013
Technology
3
420
Casual Log Collection and Querying with fluent-plugin-riak
My talk at RubyKaigi 2013
http://rubykaigi.org/2013/talk/S70
UENISHI Kota
June 01, 2013
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Metadata Management in Distributed File Systems
kuenishi
2
450
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
310
Apache Ozone behind Simulation and AI Industries
kuenishi
0
270
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.1k
A Few Ways to Accelerate Deep Learning
kuenishi
0
960
Introducing Retz
kuenishi
5
1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
650
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.2k
Mesos Frameworkの作り方 (How to Make Mesos Framework)
kuenishi
7
2.3k
Other Decks in Technology
See All in Technology
Technical Writing Meetup vol.35
soracom
PRO
2
120
Developer Experienceを向上させる基盤づくりの取り組み事例集
coconala_engineer
0
150
突撃! 隣のAmazon Bedrockユーザー 〜YouはどうしてAWSで?〜
minorun365
PRO
3
390
テスト”ケース”駆動開発 で手戻りをなくそう
ryohma0510
0
320
事前準備が肝!AI活用のための業務改革
layerx
PRO
1
380
なぜクラウドサービスで Web コンソールを提供するのか
shuta13
4
2k
Swift Testingのconfirmationを コードリーディング/Dive into Swift Testing confirmation
laprasdrum
2
260
可視化により内部品質をあげるAIドキュメントリバース/20240910 Hiromitsu Akiba
shift_evolve
0
220
JEP 480: Structured Concurrency
aya_ebata
0
130
技術的負債解消の取り組みと専門チームのお話
bengo4com
0
330
サプライチェーン攻撃に備える
ryunen344
0
280
20240911_New_Relicダッシュボード活用例
speakerdeckfk
0
110
Featured
See All Featured
What the flash - Photography Introduction
edds
67
11k
For a Future-Friendly Web
brad_frost
174
9.3k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
354
29k
Visualization
eitanlees
142
15k
It's Worth the Effort
3n
182
27k
The Pragmatic Product Professional
lauravandoore
31
6.2k
Producing Creativity
orderedlist
PRO
340
39k
The Cult of Friendly URLs
andyhume
76
6k
RailsConf 2023
tenderlove
28
810
Embracing the Ebb and Flow
colly
83
4.4k
Fantastic passwords and where to find them - at NoRuKo
philnash
48
2.8k
In The Pink: A Labor of Love
frogandcode
139
22k
Transcript
Casual Log Collection and Querying with fluent-plugin-riak @kuenishi from @basho
2013/6/1 RubyKaigi
Who the hell are you? •UENISHI, Kota (@kuenishi) •Basho Japan
KK •devoted to Distributed Systems for ~6 yrs •msgpack-erlang, Jubatus
Casual Log Collection •Aggregate Every Log with Fluentd •Put Them
all into <Some Storage You Like> •Ask your Query to <Some Storage You Like>
Whole Sketch
fluentd: casual log collector http://www.flickr.com/photos/markchadwick/8757802771/ http://www.flickr.com/photos/usdagov/5681152426/ before: logs are scattered
all over the servers in chaos after: all logs flows cleanly via fluentd in order
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing Riak
what’s ? •Distributed Key-Value Store •Focused on •Availability •Scalability •Easy
Operation, ҆ (Sleep)
when Riak? •Hadoop is too much •MongoDB is too small
•Document DB aspect of Riak •put them all into Riak
Not Only KVS •Aspect of Document Database •MapReduce in JavaScript
/ Erlang
Buy it if interested
fluent-plugin-riak JSON
fluent.conf <match apache.**> type riak # define the cluster via
pb ports nodes 192.168.0.1:8087 192.168.0.2:8087 </match>
log everything as JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200",
"size":"0", "referer":"", "agent":"", "time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
How to Query
Ruby Cluent for Querying irb> q = client.bucket(‘fluentlog’) irb> q
= q.map(“function(v){ return [v]; }”).reduce(“function(values){ return values; }“, :keep => false) irb> r = q.run()
Debug distributed JS http://www.flickr.com/photos/heatsink/110859301/
Any Other Rubyish way? http://www.flickr.com/photos/snazzyshot/5366645175/
ripple
github.com/basho/ripple •a rich Ruby toolkit for Riak, consists of •Riak
client •Riak-sessions •Ripple
http://www.flickr.com/photos/toco/2612055052/
None
None
Mohair: Not Only NoSQL http://www.flickr.com/photos/frank-wouters/2464743512/
JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200", "size":"0", "referer":"", "agent":"",
"time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
SQL create table apachelogs { host varchar(16), user varchar(256), method
varchar(5), path varchar(1024), code integer, size integer, referer text, agent varchar(1024), time timestamp, tag varchar(1024) }
“Mohair” for Querying > select * from fluentlog \ where
method = “GET” group by host
Converting SQL to MapReduce •SQL -(parslet)-> JS -> Riak mapred
•where sentence is at Map •group by, count(-) is at Reduce
Chef’s Capricious Roadmap •Secondary Index Support •Query Optimization •types: timestamp,
float •nested columns •insert / delete
check it out! github: basho/riak kuenishi/fluent-plugin-riak kuenishi/mohair (kuenishi/fluent-logger-erlang)
Conclusion •NoSQL is not NoSQL any more •put’em all into
Riak via Fluentd •Query via SQL with Mohair •waiting for pull requests
Questions? •
[email protected]
•Riak Meetup (7/10) •Riak SCR (twice in a
month) •ιϑτΣΞσβΠϯ7݄߸(nginx/riak) •σʔλϕʔεΤϯδχΞཆಡຊ