Alternative Fluentd implementation in Go

http://github.com/moriyoshi/ik

Transcript

1. A l t e r n a t i v e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
2. id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena Co., Ltd.
 • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
3. fluentd in Hatena
4. Go Programming Language
5. Go Programming Language • 2009 released • Rob Pike, Ken Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
6. about Go #1 • Compiler language • Static Typed with type inference • Garbage Collection • Concurrency model (goroutine, channel)
7. about Go #2 • High performance • Small footprint • Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
8. Performance of Go • REST API server benchmark • Faster than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
9. Major products by Go • Docker • Packer • Devops guys like Go
10. Go v1.2 • released on Dec 2013 • Pre-emption in the scheduler • Performance improvement • Library updates
11. Fluentd Go implementation • “Proof of Concept” level • developed by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
12. ik
13. Current status • Suit for leaf nodes • Support only less plugins • in_forward, out_forward, out_stdout • only one executable file
14. Configuration • support v10 conf file. • v11?? <source>      type  forward      port  24224   </source>   ! <match  **>      type  stdout   </match>  
15. Plugin system • Need to build whole plugins • Need register codes for plugins engine  :=  ik.NewEngine(logger,  scoreKeeper)   engine.RegisterInputFactory(plugins.GetForwardInputFactory())   engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory())   engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
16. in_forward • treat incoming connection with goroutine func  (in  *ForwardInput)  Start()  error  {      in.running  =  true  //  XXX:  RACE      go  func  ()  {          for  in.running  {              conn,  err  :=  in.listener.Accept()              if  err  !=  nil  {                  in.logger.Fatal(err.Error())                  continue              }              go  newForwardClient(in,  in.logger,  conn,  in.codec).handle()          }      }()      return  nil   }
17. out_forward • Periodically flush with goroutine and channel func  (output  *ForwardOutput)  run_flush(interval  int)  {     ticker  :=  time.NewTicker(time.Duration(interval)  *  time.Second)     go  func()  {       for  {         select  {         case  <-­‐  ticker.C:           output.flush()         }       }     }()   }  
18. Installation • Quite easy • wget http://repository/bin/ik
19. footprint comparison (trivial) • RSS size just after execution • with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
20. Performance • “ik” is UNDER development • not tuned for performance yet • has primitive buffering (bytes/buffer)
21. Conclusion • “ik” alternative implementation of fluentd by Go • small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik