Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Data Visualization in the Trenches
Search
Abe Stanway
May 17, 2013
Programming
5
660
Data Visualization in the Trenches
This talk was given at Bocoup's OpenVis Conf in Boston.
Abe Stanway
May 17, 2013
Tweet
Share
More Decks by Abe Stanway
See All by Abe Stanway
Building Data Driven Organizations
astanway
1
200
MOM! My algorithms SUCK
astanway
15
2.7k
A Deep Dive into Monitoring with Skyline
astanway
6
1.8k
Bring the Noise: Continuously Deploying Under a Hailstorm of Metrics
astanway
34
7.8k
Gifs as Language
astanway
2
730
Your API is a Product
astanway
3
920
Zen and the Art of Writing Commit Logs
astanway
3
760
Other Decks in Programming
See All in Programming
offers_20241022_imakiire.pdf
imakurusu
2
360
色々なIaCツールを実際に触って比較してみる
iriikeita
0
270
推し活としてのrails new/oshikatsu_ha_iizo
sakahukamaki
3
1.7k
PagerDuty を軸にした On-Call 構築と運用課題の解決 / PagerDuty Japan Community Meetup 4
horimislime
1
110
Vue3の一歩踏み込んだパフォーマンスチューニング2024
hal_spidernight
3
3.1k
PHP でアセンブリ言語のように書く技術
memory1994
PRO
1
150
【Kaigi on Rails 2024】YOUTRUST スポンサーLT
krpk1900
1
250
Sidekiqで実現する 長時間非同期処理の中断と再開 / Pausing and Resuming Long-Running Asynchronous Jobs with Sidekiq
hypermkt
6
2.7k
ピラミッド、アイスクリームコーン、SMURF: 自動テストの最適バランスを求めて / Pyramid Ice-Cream-Cone and SMURF
twada
PRO
9
1k
カラム追加で増えるActiveRecordのメモリサイズ イメージできますか?
asayamakk
4
1.6k
役立つログに取り組もう
irof
26
8.7k
Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines
sucitw
1
200
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
1
40
KATA
mclloyd
29
13k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
14
1.9k
[RailsConf 2023] Rails as a piece of cake
palkan
51
4.9k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
328
21k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Unsuck your backbone
ammeep
668
57k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
3
370
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Designing for humans not robots
tammielis
249
25k
Transcript
Abe Stanway @abestanway Data visualization in the trenches
None
1.5 Billion page views $117 Million of goods sold 950
thousand users
1.5 Billion page views $117 Million of goods sold 950
thousand users (in December)
Text We practice continuous deployment.
de • ploy /diˈploi/ Verb To release your code for
the world to see, hopefully without breaking the Internet
250+ committers, everyone deploys.
Day one: Deploy.
None
30+ DEPLOYS A DAY
Text “30 deploys a day? Is that safe?”
Text Yes, with the proper tooling.
Text Every engineer must have a finger on the pulse
of the system.
Text How do you make an entire web stack “consumable”
to a handful of engineers?
Text More information Quickly consumable More abstraction
p
None
Text Real time error logging
Text No abstraction. Fluffy information. Easy to consume.
“Not all things that break throw errors.” - Oscar Wilde
1. ssh to server 2. poke around for the log
files 3. try to remember what they mean 4. try to scroll back in time to find when they started acting up. 5. repeat
Text No abstraction. Fluffy information.
Text ...but hard to get at. Lots of friction means
not easily consumable.
Text Bump up a layer of abstraction.
1. create a graph 2. look at the graph 3.
?? 4. profit!!
StatsD
StatsD::increment(“foo.bar”)
If it moves, graph it!
If it doesn’t move, graph it anyway (it might make
a run for it)
None
Text Some abstraction. Denser information.
Text ...still hard to get at en masse.
Text Bump up a layer of abstraction.
DASHBOARDS!
Hang out with the dashboards after you push.
None
[1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20]
[1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60]
DASHBOARDS x 250000 !
None
“...but there are also unknown unknowns - there are things
we do not know we don’t know.”
Text Slightly denser information is negated by the deluge.
Text The majority remains unconsumable.
Text Bump up a layer of abstraction.
SKYLINE
A real time anomaly detection system
None
Text Very abstract, harder to understand, but the effective information
density is massive.
Text Consumption is also increased by outsourcing it to the
machine.
Text Trust becomes an issue.
So you found an anomaly.
MAYBE THERE ARE OTHERS
Oculus: a metrics correlation system
None
Text A good tool adds “touch” to the system.
Text More “touch” means more intimacy with the stack.
Text A delicate balance between insight, consumptive capacity, and actionability.
Text More abstraction leads to less intimacy...
Text ...but greater information density.
@abestanway
[email protected]
Abe Stanway Data Engineer Thanks!