Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure

Transcript

1. 1
   Using MongoDB for Logs
   05/24/2011:15:30:45PST
   Kord Campbell
   @kordless
   Wednesday, June 29, 2011
   

   View Slide

2. Loggly is a highly scalable hosted search engine for logs.
   Wednesday, June 29, 2011
   

   View Slide

3. Loggly is a highly scalable hosted search engine for logs.
   Wednesday, June 29, 2011
   

   View Slide

4. Wednesday, June 29, 2011
   

   View Slide

5. Wednesday, June 29, 2011
   

   View Slide

6. Wednesday, June 29, 2011
   

   View Slide

7. AWS Infrastructure
   syslog+HTTP
   The Loggly Stack
   Solr Cloud
   Map Reduce
   EMR!
   Wednesday, June 29, 2011
   

   View Slide

8. • Amazon’s AWS for infrastructure
   • Sylog-NG for syslog/TLS input services
   • Node.js for HTTP/HTTPs input services
   • 0MQ for event queuing & work distribution
   • S3 bucket writer for local store and archives
   • MongoDB for statistics and API /stat methods
   • Solr Cloud for scalable search and facets
   • Django/Python for middleware/app
   Loggly’s Infrastructure
   Wednesday, June 29, 2011
   

   View Slide

9. Powered by
   Wednesday, June 29, 2011
   

   View Slide

10. Powered by
    Wednesday, June 29, 2011
    

    View Slide

11. • switched because it scales well & easy to do upserts
    • runs all of Loggly’s primary stats storage
    • use master-slave replication
    • java bindings + jetty to serve aggregated data
    - use the same parsing methods for dates as solr
    - presents data the same as solr does to the middleware
    MongoDB for Statistics
    Wednesday, June 29, 2011
    

    View Slide

12. • stats job watches 0MQ for events
    • as things fly by, stuff gets counted
    • by ip, destination(port), time
    • stuff that into mongoDB
    • rollups occur every hour via upserts
    • can rebuild the entire stats store in about an hour
    MongoDB for Statistics
    Wednesday, June 29, 2011
    

    View Slide

13. MongoDB for Statistics
    {
    u'endtime': 1299948299,
    u'name': u's3countbyip',
    u'value': 1,
    u'width': 60,
    u'cust_id': u'1000',
    u'split': u'229.94.176.132',
    u'starttime': 1299948240,
    u'_id': ObjectId('4d7ba317635d28d2cdb4f337')
    }
    • MongoDB is GREAT for logging statistics
    Wednesday, June 29, 2011
    

    View Slide

14. • MongoDB is PERFECT for logging stuff if it’s semi-structured
    More Logging w/ MongoDB
    {
    "version": "1.0",
    "host": "webhead2",
    "short_message": "Short message",
    "full_message": "Backtrace here\n\nmore stuff",
    "timestamp": 1291899928,
    "level": 1,
    "facility": "payment-backend",
    "file": "/var/www/somefile.rb",
    "line": 356,
    "_user_id": 42,
    "_something_else": "foo"
    }
    Wednesday, June 29, 2011
    

    View Slide

15. • Voxify’s Robert Stewart @ http://logg.ly/tpM
    • Wrote log4mongo-java parked on Github
    More Logging w/ MongoDB
    Wednesday, June 29, 2011
    

    View Slide

16. • logstash by Jordan Sissel (Loggly’s devops extraordinaire)
    • http://logstash.net/
    • uses MongoDB for output channel
    More Logging w/ MongoDB
    Wednesday, June 29, 2011
    

    View Slide

17. • graylog2 by Lennart Koopmann (@_lennart)
    • http://graylog2.org/
    • uses MongoDB for storage and search (with ElasticSearch)
    More Logging w/ MongoDB
    Wednesday, June 29, 2011
    

    View Slide

18. • Highcharts @ http://highcharts.com/
    • Protoviz @ http://vis.stanford.edu/protovis/
    • Smoothie Charts @ http://smoothiecharts.org/
    • Google Earth plugin @ http://code.google.com/apis/earth/
    • Google Charts @ http://code.google.com/apis/chart/
    Visualization Stuff
    Wednesday, June 29, 2011
    

    View Slide

19. Free Stuff
    • Free accounts at http://loggly.com/signup
    • Free stickers
    • Free shirts
    • Free lunch
    • Free advice
    Wednesday, June 29, 2011
    

    View Slide

20. If you like beavers, you should
    work here!
    18
    Follow me
    @loggly on
    Twitter!
    http://logg.ly/jobs
    Wednesday, June 29, 2011
    

    View Slide