Upgrade to Pro — share decks privately, control downloads, hide ads and more …

At Scale, Everything is Hard

At Scale, Everything is Hard

Talk presented at dotScale in Paris. It's an exploration of what we've learned over the course of developing InfluxDB and scaling the company. I also go into our architecture for our upcoming 2.0 cloud offering that is multi-tenanted and build on top of Kubernetes.

Paul Dix

June 01, 2018
Tweet

More Decks by Paul Dix

Other Decks in Technology

Transcript

  1. Time series data is the worst and best use case

    in distributed databases dotScale 2015
  2. API

  3. UI

  4. Push Down Processing Flux Processor Storage Node Storage Node from(db:"foo")

    |> range(start:-1h) |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") |> sum() |> group() |> sort() |> limit(n:20)
  5. Push Down Processing Flux Processor Data Node Data Node from(db:"foo")

    |> range(start:-1h) |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") |> sum() |> group() |> sort() |> limit(n:20)
  6. Push Down Processing Flux Processor Data Node Data Node from(db:"foo")

    |> range(start:-1h) |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") |> sum() |> group() |> sort() |> limit(n:20)
  7. Sum 8,192 Values BenchmarkFloat64Funcs_Sum_8192-8 2000000 687 ns/op 95375.41 MB/s BenchmarkInt64Funcs_Sum_8192-8

    2000000 719 ns/op 91061.06 MB/s BenchmarkUint64Funcs_Sum_8192-8 2000000 691 ns/op 94797.29 MB/s BenchmarkFloat64Funcs_Sum_8192-8 200000 10285 ns/op 6371.41 MB/s BenchmarkInt64Funcs_Sum_8192-8 500000 3892 ns/op 16837.37 MB/s BenchmarkUint64Funcs_Sum_8192-8 500000 3929 ns/op 16680.00 MB/s AVX2 using c2goasm Pure Go
  8. Sum 8,192 Values BenchmarkFloat64Funcs_Sum_8192-8 2000000 687 ns/op 95375.41 MB/s BenchmarkInt64Funcs_Sum_8192-8

    2000000 719 ns/op 91061.06 MB/s BenchmarkUint64Funcs_Sum_8192-8 2000000 691 ns/op 94797.29 MB/s BenchmarkFloat64Funcs_Sum_8192-8 200000 10285 ns/op 6371.41 MB/s BenchmarkInt64Funcs_Sum_8192-8 500000 3892 ns/op 16837.37 MB/s BenchmarkUint64Funcs_Sum_8192-8 500000 3929 ns/op 16680.00 MB/s AVX2 using c2goasm Pure Go