Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
flowstatd
Search
Kudo Chien
July 12, 2014
Programming
1
320
flowstatd
Kudo Chien
July 12, 2014
Tweet
Share
More Decks by Kudo Chien
See All by Kudo Chien
阿迪仔, 你為什麼不寫 Unit Test @ TestCorner#14
kudochien
1
130
程式設計師的自我修養
kudochien
3
480
工程師的生涯規劃,從 React Native 開始
kudochien
11
1.2k
React Ecosystem
kudochien
0
180
如果下半輩子只想 DEBUG 怎麼辦?
kudochien
12
2.6k
如果下半輩子只想成功怎麼辦?
kudochien
2
270
Other Decks in Programming
See All in Programming
2024/11/8 関西Kaggler会 2024 #3 / Kaggle Kernel で Gemma 2 × vLLM を動かす。
kohecchi
3
290
go.mod、DockerfileやCI設定に分散しがちなGoのバージョンをまとめて管理する / Go Connect #3
arthur1
10
2.4k
Vue.js学習の振り返り
hiro_xre
2
140
ECSのサービス間通信 4つの方法を比較する 〜Canary,Blue/Greenも添えて〜
tkikuc
11
2.3k
【Kaigi on Rails 2024】YOUTRUST スポンサーLT
krpk1900
1
280
Snowflake x dbtで作るセキュアでアジャイルなデータ基盤
tsoshiro
2
460
C#/.NETのこれまでのふりかえり
tomokusaba
1
170
PLoP 2024: The evolution of the microservice architecture pattern language
cer
PRO
0
2k
Importmapを使ったJavaScriptの 読み込みとブラウザアドオンの影響
swamp09
4
1.3k
Realtime API 入門
riofujimon
0
130
GCCのプラグインを作る / I Made a GCC Plugin
shouth
1
150
Amazon Qを使ってIaCを触ろう!
maruto
0
320
Featured
See All Featured
A Philosophy of Restraint
colly
203
16k
How to train your dragon (web standard)
notwaldorf
88
5.7k
Why Our Code Smells
bkeepers
PRO
334
57k
The Pragmatic Product Professional
lauravandoore
31
6.3k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8k
4 Signs Your Business is Dying
shpigford
180
21k
Scaling GitHub
holman
458
140k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
92
16k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
23k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Transcript
flowstatd - 那五年 Kudo Chien
Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於
Trend Micro、︑Waveface! 現任 biideal CTO
Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming!
Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug
在 biideal 我們沒有辦不到的事 *誤*
flowstatd 是?
這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
The difference between genius and stupidity is that genius has
its limits.
因為有限制 才得以出類拔萃
Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port!
TOC! IP Protocol
宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script!
每⼩小時”重頭”算⼀一次統計! 是網管的災難,使⽤用者的福⾳音
宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次,網路速度越來越快,兩⼩小時可
以衝很多 GB
年少輕狂的 MySQL 時代 不管 3721,往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務!
Malicious Detection
年少輕狂的 MySQL 時代 以 CCU 全校流量來說,倒進 MySQL 平均每 ⼩小時佔⽤用 Disk
1xx MB
吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash
重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑
那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*
None
統計全宿網⼀一天的流量只需要 3.1 MB
全中正 Class B 的流量只需要 25.7 MB
Flow daemon! All in memory! Real time
Hash function v1
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Hash function v2
Architecture v1 collector! process! listen port 1025 query! process!
named pipe! socket shm Command! topN! over 5G
query! process! named pipe! socket @WanCW <(_ _)>
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Architecture v2 Single process multiplexing! kqueue / select UDP! collector
port TCP! command port 那⼀一年,我還不知道 libevent / libev
Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared
code 拆成 functions OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*
–Butler Lampson “All problems in computer science can be solved
by another level of indirection”
Object Oriented C Abstract + struct + function pointer! select()/kqueue
multiplexer! Netflow v5/v9 handlers
None
None
Usage in caller is simple
Over design 之 container_of 直接 cast 就好啦 冏
Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個
subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search
Netflow version 9 template! multiple source + multiple source id
Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/
Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用
hash 又不想⽤用太多 memory! two hash tables! source table! template table
Hash from two factors source! table template! table template! table
template! table source! table template table
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap!
對照組! pcap -> wireshark! pcap -> tcpreplay
Misc JSON input command! JSON output! cmake! logger! gzopen() /
gz*()
flowstatd-frontend 圖像化才有感,偏偏我 UI 實作能⼒力很差 ⼀一年半載又過去了! Open flash chart -> Google
Chart API! PHP -> Rails (純練習)
Release 每每回頭來看,⼀一點都不滿意,但是,好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO
fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend
因為有限制 才得以出類拔萃 不只是設計系統,run startup 也是
Image source: https://www.flickr.com/photos/vernhart/1574355240/