Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure

1
Using MongoDB for Logs
05/24/2011:15:30:45PST
Kord Campbell
@kordless
Wednesday, June 29, 2011

View Slide

Loggly is a highly scalable hosted search engine for logs.

View Slide


View Slide

AWS Infrastructure
syslog+HTTP
The Loggly Stack
Solr Cloud
Map Reduce
EMR!

View Slide

• Amazon’s AWS for infrastructure
• Sylog-NG for syslog/TLS input services
• Node.js for HTTP/HTTPs input services
• 0MQ for event queuing & work distribution
• S3 bucket writer for local store and archives
• MongoDB for statistics and API /stat methods
• Solr Cloud for scalable search and facets
• Django/Python for middleware/app
Loggly’s Infrastructure

View Slide

Powered by

View Slide

• switched because it scales well & easy to do upserts
• runs all of Loggly’s primary stats storage
• use master-slave replication
• java bindings + jetty to serve aggregated data
- use the same parsing methods for dates as solr
- presents data the same as solr does to the middleware
MongoDB for Statistics

View Slide

• stats job watches 0MQ for events
• as things fly by, stuff gets counted
• by ip, destination(port), time
• stuff that into mongoDB
• rollups occur every hour via upserts
• can rebuild the entire stats store in about an hour

View Slide

{
u'endtime': 1299948299,
u'name': u's3countbyip',
u'value': 1,
u'width': 60,
u'cust_id': u'1000',
u'split': u'229.94.176.132',
u'starttime': 1299948240,
u'_id': ObjectId('4d7ba317635d28d2cdb4f337')
}
• MongoDB is GREAT for logging statistics

View Slide

• MongoDB is PERFECT for logging stuff if it’s semi-structured
More Logging w/ MongoDB
{
"version": "1.0",
"host": "webhead2",
"short_message": "Short message",
"full_message": "Backtrace here\n\nmore stuff",
"timestamp": 1291899928,
"level": 1,
"facility": "payment-backend",
"file": "/var/www/somefile.rb",
"line": 356,
"_user_id": 42,
"_something_else": "foo"
}

View Slide

• Voxify’s Robert Stewart @ http://logg.ly/tpM
• Wrote log4mongo-java parked on Github

View Slide

• logstash by Jordan Sissel (Loggly’s devops extraordinaire)
• http://logstash.net/
• uses MongoDB for output channel

View Slide

• graylog2 by Lennart Koopmann (@_lennart)
• http://graylog2.org/
• uses MongoDB for storage and search (with ElasticSearch)

View Slide

• Highcharts @ http://highcharts.com/
• Protoviz @ http://vis.stanford.edu/protovis/
• Smoothie Charts @ http://smoothiecharts.org/
• Google Earth plugin @ http://code.google.com/apis/earth/
• Google Charts @ http://code.google.com/apis/chart/
Visualization Stuff

View Slide

Free Stuff
• Free accounts at http://loggly.com/signup
• Free stickers
• Free shirts
• Free lunch
• Free advice

View Slide

If you like beavers, you should
work here!
18
Follow me
@loggly on
Twitter!
http://logg.ly/jobs

View Slide

Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure

Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure

mongodb

More Decks by mongodb

Other Decks in Technology

Featured

Transcript