Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The new InfluxDB storage engine and some query ...
Search
Paul Dix
October 15, 2015
Technology
1
320
The new InfluxDB storage engine and some query language ideas
Short talk I gave at GranfaCon
Paul Dix
October 15, 2015
Tweet
Share
More Decks by Paul Dix
See All by Paul Dix
InfluxDB IOx Project Update - 2021-02-10
pauldix
0
230
InfluxDB IOx data lifecycle and object store persistence
pauldix
1
620
InfluxDB 2.0 and Flux
pauldix
1
730
Flux and InfluxDB 2.0
pauldix
1
1.3k
Querying Prometheus with Flux
pauldix
1
910
Flux (#fluxlang): a new (time series) data scripting language
pauldix
7
5.2k
At Scale, Everything is Hard
pauldix
2
710
IFQL and the future of InfluxData
pauldix
2
1.4k
Time series & monitoring with InfluxDB and the TICK stack
pauldix
0
470
Other Decks in Technology
See All in Technology
「現場で活躍するAIエージェント」を実現するチームと開発プロセス
tkikuchi1002
3
300
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
230
“日本一のM&A企業”を支える、少人数SREの効率化戦略 / SRE NEXT 2025
genda
1
270
AIエージェントが書くのなら直接CloudFormationを書かせればいいじゃないですか何故AWS CDKを使う必要があるのさ
watany
18
7.6k
推し書籍📚 / Books and a QA Engineer
ak1210
0
140
ClaudeCode_vs_GeminiCLI_Terraformで比較してみた
tkikuchi
1
940
QuickSight SPICE の効果的な運用戦略~S3 + Athena 構成での実践ノウハウ~/quicksight-spice-s3-athena-best-practices
emiki
0
290
今だから言えるセキュリティLT_Wordpress5.7.2未満を一斉アップデートせよ
cuebic9bic
2
170
クラウド開発の舞台裏とSRE文化の醸成 / SRE NEXT 2025 Lunch Session
kazeburo
1
580
衛星運用をソフトウェアエンジニアに依頼したときにできあがるもの
sankichi92
1
1k
ソフトウェアテストのAI活用_ver1.25
fumisuke
1
610
CDK Vibe Coding Fes
tomoki10
1
630
Featured
See All Featured
Typedesign – Prime Four
hannesfritz
42
2.7k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
KATA
mclloyd
30
14k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Automating Front-end Workflow
addyosmani
1370
200k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
830
Gamification - CAS2011
davidbonilla
81
5.4k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Transcript
The new InfluxDB storage engine and some query language ideas
Paul Dix CEO at InfluxDB @pauldix paul@influxdb.com
preliminary intro materials…
Everything is indexed by time and series
Shards 10/11/2015 10/12/2015 Data organized into Shards of time, each
is an underlying DB efficient to drop old data 10/13/2015 10/10/2015
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126 Measurement
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126 Measurement Tags
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126 Measurement Tags Fields
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126 Measurement Tags Fields Timestamp
InfluxDB data temperature,device=dev1,building=b1 internal=80,external=18 1443782126 Measurement Tags Fields Timestamp We
actually store up to ns scale timestamps but I couldn’t fit on the slide
Each series and field to a unique ID temperature,device=dev1,building=b1#internal temperature,device=dev1,building=b1#external
1 2
Data per ID is tuples ordered by time temperature,device=dev1,building=b1#internal temperature,device=dev1,building=b1#external
1 2 1 (1443782126,80) 2 (1443782126,18)
Storage Requirements
High write throughput to hundreds of thousands of series
Awesome read performance
Better Compression
Writes can’t block reads
Reads can’t block writes
Write multiple ranges simultaneously
Hot backups
Many databases open in a single process
InfluxDB’s Time Structured Merge Tree (TSM Tree)
InfluxDB’s Time Structured Merge Tree (TSM Tree) like LSM, but
different
Components WAL In memory cache Index Files
Components WAL In memory cache Index Files Similar to LSM
Trees
Components WAL In memory cache Index Files Similar to LSM
Trees Same
Components WAL In memory cache Index Files Similar to LSM
Trees Same like MemTables
Components WAL In memory cache Index Files Similar to LSM
Trees Same like MemTables like SSTables
awesome time series data WAL (an append only file)
awesome time series data WAL (an append only file) in
memory index
In Memory Cache // cache and flush variables cacheLock sync.RWMutex
cache map[string]Values flushCache map[string]Values temperature,device=dev1,building=b1#internal
In Memory Cache // cache and flush variables cacheLock sync.RWMutex
cache map[string]Values flushCache map[string]Values writes can come in while WAL flushes
// cache and flush variables cacheLock sync.RWMutex cache map[string]Values flushCache
map[string]Values dirtySort map[string]bool values can come in out of order. mark if so, sort at query time
Values in Memory type Value interface { Time() time.Time UnixNano()
int64 Value() interface{} Size() int }
awesome time series data WAL (an append only file) in
memory index on disk index (periodic flushes)
The Index Data File Min Time: 10000 Max Time: 29999
Data File Min Time: 30000 Max Time: 39999 Data File Min Time: 70000 Max Time: 99999 Contiguous blocks of time
The Index Data File Min Time: 10000 Max Time: 29999
Data File Min Time: 15000 Max Time: 39999 Data File Min Time: 70000 Max Time: 99999 can overlap
The Index cpu,host=A Min Time: 10000 Max Time: 20000 cpu,host=A
Min Time: 21000 Max Time: 39999 Data File Min Time: 70000 Max Time: 99999 but a specific series must not overlap
The Index Data File Data File Data File a file
will never overlap with more than 2 others time ascending Data File Data File
Data files are read only, like LSM SSTables
The Index Data File Min Time: 10000 Max Time: 29999
Data File Min Time: 30000 Max Time: 39999 Data File Min Time: 70000 Max Time: 99999 Data File Min Time: 10000 Max Time: 99999 they periodically get compacted (like LSM)
Compacting while appending new data
Compacting while appending new data func (w *WriteLock) LockRange(min, max
int64) { // sweet code here } func (w *WriteLock) UnlockRange(min, max int64) { // sweet code here }
Compacting while appending new data func (w *WriteLock) LockRange(min, max
int64) { // sweet code here } func (w *WriteLock) UnlockRange(min, max int64) { // sweet code here } This should block until we get it
Locking happens inside each Shard
Back to the data files… Data File Min Time: 10000
Max Time: 29999 Data File Min Time: 30000 Max Time: 39999 Data File Min Time: 70000 Max Time: 99999
Data File Layout
Data File Layout Similar to SSTables
Data File Layout
Data File Layout blocks have up to 1,000 points by
default
Data File Layout
Data File Layout 4 byte position means data files can
be at most 4GB
Data Files type dataFile struct { f *os.File size uint32
mmap []byte }
Memory mapping lets the OS handle caching for you
Compressed Data Blocks
Timestamps: encoding based on precision and deltas
Timestamps (best case): Run length encoding Deltas are all the
same for a block (only requires start time, delta, and count)
Timestamps (good case): Simple8B Ann and Moffat in "Index compression
using 64-bit words"
Timestamps (worst case): raw values nano-second timestamps with large deltas
float64: double delta Facebook’s Gorilla - google: gorilla time series
facebook https://github.com/dgryski/go-tsz
booleans are bits!
int64 uses zig-zag same as from Protobufs (adding double delta
and RLE)
string uses Snappy same compression LevelDB uses (might add dictionary
compression)
How does it perform?
Compression depends greatly on the shape of your data
Write throughput depends on batching, CPU, and memory
one test: 100,000 series 100,000 points per series 10,000,000,000 total
points 5,000 points per request c3.8xlarge, writes from 4 other systems ~390,000 points/sec ~3 bytes/point (random floats, could be better)
~400 IOPS 30%-50% CPU There’s room for improvement!
Detailed writeup https://influxdb.com/docs/v0.9/concepts/storage_engine.html
Query Language Ideas
Three different kinds of functions
Aggregates select mean(value) from cpu where host = 'A' and
time > now() - 4h group by time(5m)
Transformations select derivative(value) from cpu where host = 'A' and
time > now() - 4h group by time(5m)
Selectors select min(value) from cpu where host = 'A'; and
time > now() - 4h group by time(5m)
Then there are fills select mean(value) from cpu where host
= 'A' and time > now() - 4h group by time(5m) fill(0)
How to differentiate between the different types?
How do we chain functions together? without making breaking changes
to InfluxQL
Mix jQuery style with InfluxQL SELECT mean(value).fill(previous).derivate(1s).scale(100).as(‘mvg_avg’) FROM measurement WHERE
time > now() - 4h GROUP BY time(1m)
D3 style SELECT mean(value) .fill(previous) .derivate(1s) .scale(100) .as(‘mvg_avg’) FROM measurement
WHERE time > now() - 4h GROUP BY time(1m)
Moving the FROM? SELECT from('cpu').mean(value) from('memory').mean(value) WHERE time > now()
- 4h GROUP BY time(1m)
Moving the FROM? SELECT from('cpu').mean(value) from('memory').mean(value) WHERE time > now()
- 4h GROUP BY time(1m) consistent time and filtering applied to both
JOIN SELECT join( from('errors') .count(value), from('requests') .count(value) ).fill(0) .count(value) WHERE
time > now() - 4h GROUP BY time(1m)
Thank you! Paul Dix @pauldix paul@influxdb.com