Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Casual Log Collection and Querying with fluent-...
Search
UENISHI Kota
June 01, 2013
Technology
3
450
Casual Log Collection and Querying with fluent-plugin-riak
My talk at RubyKaigi 2013
http://rubykaigi.org/2013/talk/S70
UENISHI Kota
June 01, 2013
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Storage Systems in Preferred Networks
kuenishi
0
34
Metadata Management in Distributed File Systems
kuenishi
2
510
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
400
Apache Ozone behind Simulation and AI Industries
kuenishi
0
380
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1.1k
Introducing Retz
kuenishi
5
1.2k
Introducing Retz and how to develop practical frameworks
kuenishi
3
740
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.4k
Other Decks in Technology
See All in Technology
AWS DDoS攻撃防御の最前線
ryutakondo
1
160
【CEDEC2025】『Shadowverse: Worlds Beyond』二度目のDCG開発でゲームをリデザインする~遊びやすさと競技性の両立~
cygames
PRO
1
370
AI関数が早くなったので試してみよう
kumakura
0
300
【OptimizationNight】数理最適化のラストワンマイルとしてのUIUX
brainpadpr
2
480
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
3
13k
Segment Anything Modelの最新動向:SAM2とその発展系
tenten0727
0
780
【新卒研修資料】数理最適化 / Mathematical Optimization
brainpadpr
27
13k
MCP認可の現在地と自律型エージェント対応に向けた課題 / MCP Authorization Today and Challenges to Support Autonomous Agents
yokawasa
5
2.4k
LTに影響を受けてテンプレリポジトリを作った話
hol1kgmg
0
370
マルチプロダクト×マルチテナントを支えるモジュラモノリスを中心としたアソビューのアーキテクチャ
disc99
1
560
o11yツールを乗り換えた話
tak0x00
2
1.4k
Kiroでインフラ要件定義~テスト を実施してみた
nagisa53
3
360
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.3k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
183
54k
What's in a price? How to price your products and services
michaelherold
246
12k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
Code Reviewing Like a Champion
maltzj
524
40k
Visualization
eitanlees
146
16k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
A better future with KSS
kneath
239
17k
The Cult of Friendly URLs
andyhume
79
6.5k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Docker and Python
trallard
45
3.5k
How to train your dragon (web standard)
notwaldorf
96
6.2k
Transcript
Casual Log Collection and Querying with fluent-plugin-riak @kuenishi from @basho
2013/6/1 RubyKaigi
Who the hell are you? •UENISHI, Kota (@kuenishi) •Basho Japan
KK •devoted to Distributed Systems for ~6 yrs •msgpack-erlang, Jubatus
Casual Log Collection •Aggregate Every Log with Fluentd •Put Them
all into <Some Storage You Like> •Ask your Query to <Some Storage You Like>
Whole Sketch
fluentd: casual log collector http://www.flickr.com/photos/markchadwick/8757802771/ http://www.flickr.com/photos/usdagov/5681152426/ before: logs are scattered
all over the servers in chaos after: all logs flows cleanly via fluentd in order
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing Riak
what’s ? •Distributed Key-Value Store •Focused on •Availability •Scalability •Easy
Operation, ҆ (Sleep)
when Riak? •Hadoop is too much •MongoDB is too small
•Document DB aspect of Riak •put them all into Riak
Not Only KVS •Aspect of Document Database •MapReduce in JavaScript
/ Erlang
Buy it if interested
fluent-plugin-riak JSON
fluent.conf <match apache.**> type riak # define the cluster via
pb ports nodes 192.168.0.1:8087 192.168.0.2:8087 </match>
log everything as JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200",
"size":"0", "referer":"", "agent":"", "time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
How to Query
Ruby Cluent for Querying irb> q = client.bucket(‘fluentlog’) irb> q
= q.map(“function(v){ return [v]; }”).reduce(“function(values){ return values; }“, :keep => false) irb> r = q.run()
Debug distributed JS http://www.flickr.com/photos/heatsink/110859301/
Any Other Rubyish way? http://www.flickr.com/photos/snazzyshot/5366645175/
ripple
github.com/basho/ripple •a rich Ruby toolkit for Riak, consists of •Riak
client •Riak-sessions •Ripple
http://www.flickr.com/photos/toco/2612055052/
None
None
Mohair: Not Only NoSQL http://www.flickr.com/photos/frank-wouters/2464743512/
JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200", "size":"0", "referer":"", "agent":"",
"time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
SQL create table apachelogs { host varchar(16), user varchar(256), method
varchar(5), path varchar(1024), code integer, size integer, referer text, agent varchar(1024), time timestamp, tag varchar(1024) }
“Mohair” for Querying > select * from fluentlog \ where
method = “GET” group by host
Converting SQL to MapReduce •SQL -(parslet)-> JS -> Riak mapred
•where sentence is at Map •group by, count(-) is at Reduce
Chef’s Capricious Roadmap •Secondary Index Support •Query Optimization •types: timestamp,
float •nested columns •insert / delete
check it out! github: basho/riak kuenishi/fluent-plugin-riak kuenishi/mohair (kuenishi/fluent-logger-erlang)
Conclusion •NoSQL is not NoSQL any more •put’em all into
Riak via Fluentd •Query via SQL with Mohair •waiting for pull requests
Questions? •
[email protected]
•Riak Meetup (7/10) •Riak SCR (twice in a
month) •ιϑτΣΞσβΠϯ7݄߸(nginx/riak) •σʔλϕʔεΤϯδχΞཆಡຊ