Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Data Driven Organizations
Search
Abe Stanway
September 13, 2014
Programming
1
210
Building Data Driven Organizations
Given at IT Weekend 2014 in Kiev
Abe Stanway
September 13, 2014
Tweet
Share
More Decks by Abe Stanway
See All by Abe Stanway
MOM! My algorithms SUCK
astanway
15
2.8k
A Deep Dive into Monitoring with Skyline
astanway
6
1.8k
Bring the Noise: Continuously Deploying Under a Hailstorm of Metrics
astanway
34
8k
Data Visualization in the Trenches
astanway
5
720
Gifs as Language
astanway
2
850
Your API is a Product
astanway
3
980
Zen and the Art of Writing Commit Logs
astanway
3
830
Other Decks in Programming
See All in Programming
CSC509 Lecture 03
javiergs
PRO
0
320
GraphQL×Railsアプリのデータベース負荷分散 - 月間3,000万人利用サービスを無停止で
koxya
1
1k
プログラミングどうやる? ~テスト駆動開発から学ぶ達人の型~
a_okui
0
190
iOSDC.pdf
chronos2500
2
650
CSS Linter の現在地 2025年のベストプラクティスを探る
ryo_manba
10
3.2k
議事録の要点整理を自動化! サーバレス Bot 構築術
penpeen
3
1.6k
2分台で1500examples完走!爆速CIを支える環境構築術 - Kaigi on Rails 2025
falcon8823
3
2.8k
気づいて!アプリからのSOS 〜App Store Connect APIで始めるパフォーマンス健康診断〜
waka12
0
260
あなたの知らない「動画広告」の世界 - iOSDC Japan 2025
ukitaka
0
340
Playwrightはどのようにクロスブラウザをサポートしているのか
yotahada3
7
2.2k
(Extension DC 2025) Actor境界を越える技術
teamhimeh
1
200
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
4
1.2k
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
140
34k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.1k
Site-Speed That Sticks
csswizardry
11
870
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
890
Why Our Code Smells
bkeepers
PRO
339
57k
Embracing the Ebb and Flow
colly
88
4.8k
Documentation Writing (for coders)
carmenintech
75
5k
The Pragmatic Product Professional
lauravandoore
36
6.9k
Thoughts on Productivity
jonyablonski
70
4.8k
Transcript
@AbeStanway BUILDING A DATA DRIVEN ORGANIZATION
1. why 2. how
1. why 2. how
“DATA IS THE NEW GOLD”
Predict the future!
Retain Customers!
Grow the business!
Recommend content!
Drive Engagement!
unclear paths to $$$
IN IT, It’s clear.
Data are Dollars
. IT Working = +$$$ IT Not Working = -$$$
. .
How do you know if your IT is working right
now?
How do you know if you are earning money right
now?
KPIs. What are they?
Etsy: Literally a Money per second Graph
Planet Labs: Literally an Images per day graph
What are the Kpis for kips?
$ per second items bought per second page requests per
second database queries per second memcache hits per second fread() per second
If you do not have the data about your infrastructure,
it is already broken. LEsson:
None
Test driven development -> data driven development
Without data, you are flying blind
How do you know you’re hitting your goals?
How do you know if You’re making the right ones
in the first place?
How do you know if you’re still in business?
How do you even know what planet you live on?
Assumptions are death
You need data, yo.
1. why 2. how
1. collect 2. analyze 3. ??? 4. Profit!
1. collect 2. analyze 3. ACT 4. Profit!
data that cannot be acted upon should not be analyzed.
None
You are running a business, not an art museum
You are Trying to Win the market, not a fields
medal
This can be disappointing
Data SCientist?
Data Scientist? Realist.
Find a way to Align your employees intellectual curiosity With
your Real business needs. LEssoN:
Train your organization
you need a data culture.
“It’s not shipped until it’s monitored”
“If you are not looking at Dashboards, you are not
doing your job”
Building instrumentation and watching dashboards are hard And Time consuming
App code -> statsD -> Graphite -> Dashboards -> Insights
by hand by hand by hand by hand by hand
Developers just want to code
Let’s automate
Which is easier to automate? Insights or data collection?
Insights are sexy and fun
Collection is hard And unsexy
Collection is hard And Boring
Collection is hard And unsexy
We’re on track to have excellent automated insights
anomaly detection
App code -> statsD -> Graphite -> Dashboards -> Insights
by hand by hand by hand AUTOMATIC! AUTOMATIC!
(…if only we had the data)
How do we automate data collection?
currently have ganglia, New relic, collectD, etc
NOT WHAT WE NEED
they provide data about your raw machines, not your CUSTOM
DEVELOPED TECHNOLOGY And Application level logic
Healthy servers don’t make you money. Healthy services do.
enter LARIMAR
Full disclosure: this is my new PROJECT ! we’re going
to talk about it because i’m pretty excited and the beta is opening up soon.
LARIMAR uses raw machine metrics to infer App level architecture
and inform developers about problems
A service: cpu resources disk io PCAP data ports Used
syscalls
A service: cpu resources disk io PCAP data ports Used
syscalls service fingerprint MACHINE LEARNING
A service: cpu resources disk io PCAP data ports Used
syscalls ABNORMAL BEHAVIOR MACHINE LEARNING
a system: service service service service service MACHINE LEARNING graphical
system fingerprint
a system: service service service service service MACHINE LEARNING Abnormal,
holistic system behavior
Larimar automates both analysis And Relevant data collection
so your developers can focus on coding and acting on
insights
No configuration!
App code -> statsD -> Graphite -> Dashboards -> Insights
by hand AUTOMATIC! AUTOMATIC! AUTOMATIC! AUTOMATIC!
1. collect 2. analyze 3. ACT 4. Profit!
organizational shifts are still needed to inspire ACTION on Data
but ACTION is easier to inspire when there is lots
of data and lots of insight everywhere
Create a culture where your developers create these kinds of
tools
When a data driven mindset is the default, tools will
build themselves.
Thanks! @abestanway ! ! larimar.io @larimarhq