Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
flowstatd
Search
Kudo Chien
July 12, 2014
Programming
1
320
flowstatd
Kudo Chien
July 12, 2014
Tweet
Share
More Decks by Kudo Chien
See All by Kudo Chien
阿迪仔, 你為什麼不寫 Unit Test @ TestCorner#14
kudochien
1
130
程式設計師的自我修養
kudochien
3
480
工程師的生涯規劃,從 React Native 開始
kudochien
11
1.2k
React Ecosystem
kudochien
0
180
如果下半輩子只想 DEBUG 怎麼辦?
kudochien
12
2.6k
如果下半輩子只想成功怎麼辦?
kudochien
2
270
Other Decks in Programming
See All in Programming
Piniaの現状と今後
waka292
5
1.5k
推し活としてのrails new/oshikatsu_ha_iizo
sakahukamaki
3
1.7k
CSC305 Lecture 13
javiergs
PRO
0
130
EventSourcingの理想と現実
wenas
6
2.1k
[PyCon Korea 2024 Keynote] 커뮤니티와 파이썬, 그리고 우리
beomi
0
110
ピラミッド、アイスクリームコーン、SMURF: 自動テストの最適バランスを求めて / Pyramid Ice-Cream-Cone and SMURF
twada
PRO
9
1k
cXML という電子商取引の トランザクションを支える プロトコルと向きあっている話
phigasui
3
2.3k
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
7
2.8k
のびしろを広げる巻き込まれ力:偶然を活かすキャリアの作り方/oso2024
takahashiikki
1
410
Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines
sucitw
1
200
JaSST 24 九州:ワークショップ(は除く)実践!マインドマップを活用したソフトウェアテスト+活用事例
satohiroyuki
0
260
カスタムしながら理解するGraphQL Connection
yanagii
1
1.2k
Featured
See All Featured
Automating Front-end Workflow
addyosmani
1365
200k
How GitHub (no longer) Works
holman
311
140k
The Language of Interfaces
destraynor
154
24k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
Optimizing for Happiness
mojombo
376
69k
Designing Experiences People Love
moore
138
23k
How to train your dragon (web standard)
notwaldorf
88
5.7k
Why You Should Never Use an ORM
jnunemaker
PRO
53
9k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
Fashionably flexible responsive web design (full day workshop)
malarkey
404
65k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
Transcript
flowstatd - 那五年 Kudo Chien
Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於
Trend Micro、︑Waveface! 現任 biideal CTO
Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming!
Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug
在 biideal 我們沒有辦不到的事 *誤*
flowstatd 是?
這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
The difference between genius and stupidity is that genius has
its limits.
因為有限制 才得以出類拔萃
Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port!
TOC! IP Protocol
宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script!
每⼩小時”重頭”算⼀一次統計! 是網管的災難,使⽤用者的福⾳音
宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次,網路速度越來越快,兩⼩小時可
以衝很多 GB
年少輕狂的 MySQL 時代 不管 3721,往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務!
Malicious Detection
年少輕狂的 MySQL 時代 以 CCU 全校流量來說,倒進 MySQL 平均每 ⼩小時佔⽤用 Disk
1xx MB
吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash
重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑
那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*
None
統計全宿網⼀一天的流量只需要 3.1 MB
全中正 Class B 的流量只需要 25.7 MB
Flow daemon! All in memory! Real time
Hash function v1
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Hash function v2
Architecture v1 collector! process! listen port 1025 query! process!
named pipe! socket shm Command! topN! over 5G
query! process! named pipe! socket @WanCW <(_ _)>
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Architecture v2 Single process multiplexing! kqueue / select UDP! collector
port TCP! command port 那⼀一年,我還不知道 libevent / libev
Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared
code 拆成 functions OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*
–Butler Lampson “All problems in computer science can be solved
by another level of indirection”
Object Oriented C Abstract + struct + function pointer! select()/kqueue
multiplexer! Netflow v5/v9 handlers
None
None
Usage in caller is simple
Over design 之 container_of 直接 cast 就好啦 冏
Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個
subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search
Netflow version 9 template! multiple source + multiple source id
Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/
Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用
hash 又不想⽤用太多 memory! two hash tables! source table! template table
Hash from two factors source! table template! table template! table
template! table source! table template table
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap!
對照組! pcap -> wireshark! pcap -> tcpreplay
Misc JSON input command! JSON output! cmake! logger! gzopen() /
gz*()
flowstatd-frontend 圖像化才有感,偏偏我 UI 實作能⼒力很差 ⼀一年半載又過去了! Open flash chart -> Google
Chart API! PHP -> Rails (純練習)
Release 每每回頭來看,⼀一點都不滿意,但是,好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO
fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend
因為有限制 才得以出類拔萃 不只是設計系統,run startup 也是
Image source: https://www.flickr.com/photos/vernhart/1574355240/