Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.5k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
560
Static Code Analysis Using SSA
benbjohnson
6
1.8k
Writing a High Performance Database in Go
benbjohnson
11
6.8k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
105
28k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
2k
New Tech - Sky
benbjohnson
2
380
Sky Analytics Pitch Deck
benbjohnson
0
620
Other Decks in Technology
See All in Technology
生成AI時代のPythonセキュリティとガバナンス
abenben
0
140
AWS DMS で SQL Server を移行してみた/aws-dms-sql-server-migration
emiki
0
250
Oracle Database@Google Cloud:サービス概要のご紹介
oracle4engineer
PRO
0
380
IoTLT@ストラタシスジャパン_20251021
norioikedo
0
140
可観測性は開発環境から、開発環境にもオブザーバビリティ導入のススメ
layerx
PRO
4
1.6k
現場の壁を乗り越えて、 「計装注入」が拓く オブザーバビリティ / Beyond the Field Barriers: Instrumentation Injection and the Future of Observability
aoto
PRO
1
640
スタートアップの現場で実践しているテストマネジメント #jasst_kyushu
makky_tyuyan
0
140
AIエージェントによる業務効率化への飽くなき挑戦-AWS上の実開発事例から学んだ効果、現実そしてギャップ-
nasuvitz
5
1.4k
Building a cloud native business on open source
lizrice
0
190
頭部ふわふわ浄酔器
uyupun
0
230
Amazon Athena で JSON・Parquet・Iceberg のデータを検索し、性能を比較してみた
shigeruoda
1
150
Observability — Extending Into Incident Response
nari_ex
1
540
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
431
66k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
Typedesign – Prime Four
hannesfritz
42
2.8k
Gamification - CAS2011
davidbonilla
81
5.5k
jQuery: Nuts, Bolts and Bling
dougneiner
65
7.9k
A Modern Web Designer's Workflow
chriscoyier
697
190k
What's in a price? How to price your products and services
michaelherold
246
12k
BBQ
matthewcrist
89
9.9k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
31
2.7k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.7k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]