Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Alternative Fluentd implementation in Go

Shinji Tanaka
December 13, 2013

Alternative Fluentd implementation in Go

Shinji Tanaka

December 13, 2013
Tweet

More Decks by Shinji Tanaka

Other Decks in Technology

Transcript

  1. A l t e r n a t i v

    e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
  2. id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena

    Co., Ltd.
 • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
  3. Go Programming Language • 2009 released • Rob Pike, Ken

    Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
  4. about Go #1 • Compiler language • Static Typed with

    type inference • Garbage Collection • Concurrency model (goroutine, channel)
  5. about Go #2 • High performance • Small footprint •

    Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
  6. Performance of Go • REST API server benchmark • Faster

    than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
  7. Go v1.2 • released on Dec 2013 • Pre-emption in

    the scheduler • Performance improvement • Library updates
  8. Fluentd Go implementation • “Proof of Concept” level • developed

    by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
  9. ik

  10. Current status • Suit for leaf nodes • Support only

    less plugins • in_forward, out_forward, out_stdout • only one executable file
  11. Configuration • support v10 conf file. • v11?? <source>  

       type  forward      port  24224   </source>   ! <match  **>      type  stdout   </match>  
  12. Plugin system • Need to build whole plugins • Need

    register codes for plugins engine  :=  ik.NewEngine(logger,  scoreKeeper)   engine.RegisterInputFactory(plugins.GetForwardInputFactory())   engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory())   engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
  13. in_forward • treat incoming connection with goroutine func  (in  *ForwardInput)

     Start()  error  {      in.running  =  true  //  XXX:  RACE      go  func  ()  {          for  in.running  {              conn,  err  :=  in.listener.Accept()              if  err  !=  nil  {                  in.logger.Fatal(err.Error())                  continue              }              go  newForwardClient(in,  in.logger,  conn,  in.codec).handle()          }      }()      return  nil   }
  14. out_forward • Periodically flush with goroutine and channel func  (output

     *ForwardOutput)  run_flush(interval  int)  {     ticker  :=  time.NewTicker(time.Duration(interval)  *  time.Second)     go  func()  {       for  {         select  {         case  <-­‐  ticker.C:           output.flush()         }       }     }()   }  
  15. footprint comparison (trivial) • RSS size just after execution •

    with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
  16. Performance • “ik” is UNDER development • not tuned for

    performance yet • has primitive buffering (bytes/buffer)
  17. Conclusion • “ik” alternative implementation of fluentd by Go •

    small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik