Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Casual Log Collection and Querying with fluent-...

Casual Log Collection and Querying with fluent-plugin-riak

My talk at RubyKaigi 2013 http://rubykaigi.org/2013/talk/S70

UENISHI Kota

June 01, 2013
Tweet

More Decks by UENISHI Kota

Other Decks in Technology

Transcript

  1. Who the hell are you? •UENISHI, Kota (@kuenishi) •Basho Japan

    KK •devoted to Distributed Systems for ~6 yrs •msgpack-erlang, Jubatus
  2. Casual Log Collection •Aggregate Every Log with Fluentd •Put Them

    all into <Some Storage You Like> •Ask your Query to <Some Storage You Like>
  3. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache

    Frontend Access logs syslogd App logs System logs Backend Databases
  4. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache

    Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
  5. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache

    Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing Riak
  6. when Riak? •Hadoop is too much •MongoDB is too small

    •Document DB aspect of Riak •put them all into Riak
  7. fluent.conf <match apache.**> type riak # define the cluster via

    pb ports nodes 192.168.0.1:8087 192.168.0.2:8087 </match>
  8. log everything as JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200",

    "size":"0", "referer":"", "agent":"", "time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
  9. Ruby Cluent for Querying irb> q = client.bucket(‘fluentlog’) irb> q

    = q.map(“function(v){ return [v]; }”).reduce(“function(values){ return values; }“, :keep => false) irb> r = q.run()
  10. SQL create table apachelogs { host varchar(16), user varchar(256), method

    varchar(5), path varchar(1024), code integer, size integer, referer text, agent varchar(1024), time timestamp, tag varchar(1024) }
  11. Converting SQL to MapReduce •SQL -(parslet)-> JS -> Riak mapred

    •where sentence is at Map •group by, count(-) is at Reduce
  12. Conclusion •NoSQL is not NoSQL any more •put’em all into

    Riak via Fluentd •Query via SQL with Mohair •waiting for pull requests
  13. Questions? •[email protected] •Riak Meetup (7/10) •Riak SCR (twice in a

    month) •ιϑτ΢ΣΞσβΠϯ7݄߸(nginx/riak) •σʔλϕʔεΤϯδχΞཆ੒ಡຊ