Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Alternative Fluentd implementation in Go
Search
Shinji Tanaka
December 13, 2013
Technology
3
6k
Alternative Fluentd implementation in Go
http://github.com/moriyoshi/ik
Shinji Tanaka
December 13, 2013
Tweet
Share
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
2.9k
How to Contribute Go
stanaka
4
18k
Introduction to Mackerel
stanaka
0
1k
Upcoming features in 2016, Mackerel
stanaka
0
3.3k
Mackerel's first year and onward
stanaka
0
2.9k
Mackerel's first year and onward (in English)
stanaka
0
370
Blue-Green Deployment with ECS and monitoring
stanaka
0
580
Mackerel Meetup #4 (in Eng)
stanaka
0
480
Mackerel Meetup #4
stanaka
0
3.1k
Other Decks in Technology
See All in Technology
なぜ今 AI Agent なのか _近藤憲児
kenjikondobai
4
1.4k
New Relicを活用したSREの最初のステップ / NRUG OKINAWA VOL.3
isaoshimizu
3
630
OCI Network Firewall 概要
oracle4engineer
PRO
0
4.2k
適材適所の技術選定 〜GraphQL・REST API・tRPC〜 / Optimal Technology Selection
kakehashi
1
700
CysharpのOSS群から見るModern C#の現在地
neuecc
2
3.5k
組織成長を加速させるオンボーディングの取り組み
sudoakiy
2
220
Amazon CloudWatch Network Monitor のススメ
yuki_ink
1
210
OS 標準のデザインシステムを超えて - より柔軟な Flutter テーマ管理 | FlutterKaigi 2024
ronnnnn
1
280
100 名超が参加した日経グループ横断の競技型 AWS 学習イベント「Nikkei Group AWS GameDay」の紹介/mediajaws202411
nikkei_engineer_recruiting
1
170
DynamoDB でスロットリングが発生したとき_大盛りver/when_throttling_occurs_in_dynamodb_long
emiki
1
440
日経電子版のStoreKit2フルリニューアル
shimastripe
1
140
Terraform Stacks入門 #HashiTalks
msato
0
360
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
33
2.4k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
93
16k
Teambox: Starting and Learning
jrom
133
8.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.1k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
Typedesign – Prime Four
hannesfritz
40
2.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
The Language of Interfaces
destraynor
154
24k
KATA
mclloyd
29
14k
Code Review Best Practice
trishagee
64
17k
Scaling GitHub
holman
458
140k
Transcript
A l t e r n a t i v
e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena
Co., Ltd. • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
fluentd in Hatena
Go Programming Language
Go Programming Language • 2009 released • Rob Pike, Ken
Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
about Go #1 • Compiler language • Static Typed with
type inference • Garbage Collection • Concurrency model (goroutine, channel)
about Go #2 • High performance • Small footprint •
Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
Performance of Go • REST API server benchmark • Faster
than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
Major products by Go • Docker • Packer • Devops
guys like Go
Go v1.2 • released on Dec 2013 • Pre-emption in
the scheduler • Performance improvement • Library updates
Fluentd Go implementation • “Proof of Concept” level • developed
by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
ik
Current status • Suit for leaf nodes • Support only
less plugins • in_forward, out_forward, out_stdout • only one executable file
Configuration • support v10 conf file. • v11?? <source>
type forward port 24224 </source> ! <match **> type stdout </match>
Plugin system • Need to build whole plugins • Need
register codes for plugins engine := ik.NewEngine(logger, scoreKeeper) engine.RegisterInputFactory(plugins.GetForwardInputFactory()) engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory()) engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
in_forward • treat incoming connection with goroutine func (in *ForwardInput)
Start() error { in.running = true // XXX: RACE go func () { for in.running { conn, err := in.listener.Accept() if err != nil { in.logger.Fatal(err.Error()) continue } go newForwardClient(in, in.logger, conn, in.codec).handle() } }() return nil }
out_forward • Periodically flush with goroutine and channel func (output
*ForwardOutput) run_flush(interval int) { ticker := time.NewTicker(time.Duration(interval) * time.Second) go func() { for { select { case <-‐ ticker.C: output.flush() } } }() }
Installation • Quite easy • wget http://repository/bin/ik
footprint comparison (trivial) • RSS size just after execution •
with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
Performance • “ik” is UNDER development • not tuned for
performance yet • has primitive buffering (bytes/buffer)
Conclusion • “ik” alternative implementation of fluentd by Go •
small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik