Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.4k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
530
Static Code Analysis Using SSA
benbjohnson
6
1.6k
Writing a High Performance Database in Go
benbjohnson
11
6.6k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
28k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
2k
New Tech - Sky
benbjohnson
2
370
Sky Analytics Pitch Deck
benbjohnson
0
570
Other Decks in Technology
See All in Technology
Aurora_BlueGreenDeploymentsやってみた
tsukasa_ishimaru
1
130
30万人が利用するチャットをFirebase Realtime DatabaseからActionCableへ移行する方法
ryosk7
5
350
[AWS JAPAN 生成AIハッカソン] Dialog の紹介
yoshimi0227
0
150
ガチ勢によるPipeCD運用大全〜滑らかなCI/CDを添えて〜 / ai-pipecd-encyclopedia
cyberagentdevelopers
PRO
3
210
初心者に Vue.js を 教えるには
tsukuha
5
390
【若手エンジニア応援LT会】AWSで繋がり、共に成長! ~コミュニティ活動と新人教育への挑戦~
kazushi_ohata
0
180
リンクアンドモチベーション ソフトウェアエンジニア向け紹介資料 / Introduction to Link and Motivation for Software Engineers
lmi
4
290k
【技術書典17】OpenFOAM(自宅で極める流体解析)2次元円柱まわりの流れ
kamakiri1225
0
220
Product Engineer Night #6プロダクトエンジニアを育む仕組み・施策
hacomono
PRO
1
470
よくわからんサービスについての問い合わせが来たときの強い味方 Amazon Q について
kazzpapa3
0
220
CAMERA-Suite: 広告文生成のための評価スイート / ai-camera-suite
cyberagentdevelopers
PRO
3
270
日経電子版におけるリアルタイムレコメンドシステム開発の事例紹介/nikkei-realtime-recommender-system
yng87
1
510
Featured
See All Featured
Six Lessons from altMBA
skipperchong
26
3.5k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
290
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
Making Projects Easy
brettharned
115
5.9k
The Language of Interfaces
destraynor
154
24k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
790
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
14
1.9k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
Thoughts on Productivity
jonyablonski
67
4.3k
10 Git Anti Patterns You Should be Aware of
lemiorhan
654
59k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Faster Mobile Websites
deanohume
304
30k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]