Observability_at_Google_--_OSCON.pdf

Observability at Google JBD, Google (@rakyll)

@rakyll History Long history of distributed systems 10ks of different
services built by 100s of teams Many backends/analysis tools invented here ™

@rakyll

@rakyll 100% availability (is a lie)

“ @rakyll A service is available if users cannot tell
there is an outage.

“ @rakyll Google Load Balancers are available if users cannot
tell there is an outage.

@rakyll Principled way of saying what level of downtime is
acceptable. • Error rate • Latency expectations SLOs

@rakyll An observable system tells more than its availability.

@rakyll Context, status, expectations, debuggability

@rakyll How? Observe by collecting signals Export them to analysis
tools Correlate and analyze to find root cause

@rakyll

@rakyll This is hard Must have integrations for web, RPC,
and storage clients Must support all languages Must be context aware (e.g. canary vs prod) Must support many analysis tools Developers need to add custom instrumentation

@rakyll This is too hard!

@rakyll Borg Stubby Census

opencensus.io

@rakyll

@rakyll Z-Pages • Allows processes report their own dashboards. •
Z-Pages have no sampling.

@rakyll Try! import “go.opencensus.io/plugin/ocgrpc” s := grpc.NewServer(grpc.StatsHandler(&ocgrpc.ServerHandler{})) if err :=
s.Serve(lis); err != nil { log.Fatalf("Failed to serve: %v", err) }

@rakyll import ( “go.opencensus.io/stats/view” “go.opencensus.io/trace” “contrib.go.opencensus.io/exporter/stackdriver” ) exporter, err :=
stackdriver.NewExporter(stackdriver.Options{ … }) if err != nil { log.Fatal(err) } view.RegisterExporter(exporter) trace.RegisterExporter(exporter)

@rakyll

@rakyll Roadmap Stable libraries in 8+ languages Exporter daemon Cluster-wide
Z-Pages Smart sampling Exemplars Framework, database, MQ integrations

opencensus.io

Thank you! opencensus.io JBD, Google [email protected] @rakyll

Observability_at_Google_--_OSCON.pdf

Observability_at_Google_--_OSCON.pdf

JBD

More Decks by JBD

Other Decks in Programming

Featured

Transcript

Observability at Google JBD, Google (@rakyll)

@rakyll History Long history of distributed systems 10ks of different

@rakyll

@rakyll 100% availability (is a lie)

“ @rakyll A service is available if users cannot tell

“ @rakyll Google Load Balancers are available if users cannot

@rakyll Principled way of saying what level of downtime is

@rakyll An observable system tells more than its availability.

@rakyll Context, status, expectations, debuggability

@rakyll How? Observe by collecting signals Export them to analysis

@rakyll

@rakyll

@rakyll

@rakyll

@rakyll This is hard Must have integrations for web, RPC,

@rakyll This is too hard!

@rakyll Borg Stubby Census

opencensus.io

@rakyll

@rakyll

@rakyll

@rakyll

@rakyll Z-Pages • Allows processes report their own dashboards. •

@rakyll Try! import “go.opencensus.io/plugin/ocgrpc” s := grpc.NewServer(grpc.StatsHandler(&ocgrpc.ServerHandler{})) if err :=

@rakyll import ( “go.opencensus.io/stats/view” “go.opencensus.io/trace” “contrib.go.opencensus.io/exporter/stackdriver” ) exporter, err :=

@rakyll

@rakyll

@rakyll Roadmap Stable libraries in 8+ languages Exporter daemon Cluster-wide

opencensus.io

Thank you! opencensus.io JBD, Google [email protected] @rakyll