Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
flowstatd
Search
Kudo Chien
July 12, 2014
Programming
1
320
flowstatd
Kudo Chien
July 12, 2014
Tweet
Share
More Decks by Kudo Chien
See All by Kudo Chien
阿迪仔, 你為什麼不寫 Unit Test @ TestCorner#14
kudochien
1
130
程式設計師的自我修養
kudochien
3
480
工程師的生涯規劃,從 React Native 開始
kudochien
11
1.2k
React Ecosystem
kudochien
0
180
如果下半輩子只想 DEBUG 怎麼辦?
kudochien
12
2.6k
如果下半輩子只想成功怎麼辦?
kudochien
2
270
Other Decks in Programming
See All in Programming
初めてDefinitelyTypedにPRを出した話
syumai
0
420
CSC509 Lecture 09
javiergs
PRO
0
140
とにかくAWS GameDay!AWSは世界の共通言語! / Anyway, AWS GameDay! AWS is the world's lingua franca!
seike460
PRO
1
900
『ドメイン駆動設計をはじめよう』のモデリングアプローチ
masuda220
PRO
8
540
TypeScript Graph でコードレビューの心理的障壁を乗り越える
ysk8hori
2
1.2k
型付き API リクエストを実現するいくつかの手法とその選択 / Typed API Request
euxn23
8
2.2k
Outline View in SwiftUI
1024jp
1
330
Jakarta EE meets AI
ivargrimstad
0
620
Enabling DevOps and Team Topologies Through Architecture: Architecting for Fast Flow
cer
PRO
0
340
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
9
3.3k
色々なIaCツールを実際に触って比較してみる
iriikeita
0
330
Kaigi on Rails 2024 〜運営の裏側〜
krpk1900
1
240
Featured
See All Featured
Music & Morning Musume
bryan
46
6.2k
Documentation Writing (for coders)
carmenintech
65
4.4k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
24k
For a Future-Friendly Web
brad_frost
175
9.4k
The World Runs on Bad Software
bkeepers
PRO
65
11k
4 Signs Your Business is Dying
shpigford
180
21k
Building Better People: How to give real-time feedback that sticks.
wjessup
364
19k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.7k
Optimizing for Happiness
mojombo
376
70k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
900
Transcript
flowstatd - 那五年 Kudo Chien
Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於
Trend Micro、︑Waveface! 現任 biideal CTO
Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming!
Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug
在 biideal 我們沒有辦不到的事 *誤*
flowstatd 是?
這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
The difference between genius and stupidity is that genius has
its limits.
因為有限制 才得以出類拔萃
Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port!
TOC! IP Protocol
宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script!
每⼩小時”重頭”算⼀一次統計! 是網管的災難,使⽤用者的福⾳音
宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次,網路速度越來越快,兩⼩小時可
以衝很多 GB
年少輕狂的 MySQL 時代 不管 3721,往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務!
Malicious Detection
年少輕狂的 MySQL 時代 以 CCU 全校流量來說,倒進 MySQL 平均每 ⼩小時佔⽤用 Disk
1xx MB
吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash
重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑
那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*
None
統計全宿網⼀一天的流量只需要 3.1 MB
全中正 Class B 的流量只需要 25.7 MB
Flow daemon! All in memory! Real time
Hash function v1
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Hash function v2
Architecture v1 collector! process! listen port 1025 query! process!
named pipe! socket shm Command! topN! over 5G
query! process! named pipe! socket @WanCW <(_ _)>
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Architecture v2 Single process multiplexing! kqueue / select UDP! collector
port TCP! command port 那⼀一年,我還不知道 libevent / libev
Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared
code 拆成 functions OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*
–Butler Lampson “All problems in computer science can be solved
by another level of indirection”
Object Oriented C Abstract + struct + function pointer! select()/kqueue
multiplexer! Netflow v5/v9 handlers
None
None
Usage in caller is simple
Over design 之 container_of 直接 cast 就好啦 冏
Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個
subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search
Netflow version 9 template! multiple source + multiple source id
Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/
Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用
hash 又不想⽤用太多 memory! two hash tables! source table! template table
Hash from two factors source! table template! table template! table
template! table source! table template table
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap!
對照組! pcap -> wireshark! pcap -> tcpreplay
Misc JSON input command! JSON output! cmake! logger! gzopen() /
gz*()
flowstatd-frontend 圖像化才有感,偏偏我 UI 實作能⼒力很差 ⼀一年半載又過去了! Open flash chart -> Google
Chart API! PHP -> Rails (純練習)
Release 每每回頭來看,⼀一點都不滿意,但是,好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO
fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend
因為有限制 才得以出類拔萃 不只是設計系統,run startup 也是
Image source: https://www.flickr.com/photos/vernhart/1574355240/