Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.5k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
540
Static Code Analysis Using SSA
benbjohnson
6
1.7k
Writing a High Performance Database in Go
benbjohnson
11
6.7k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
28k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
2k
New Tech - Sky
benbjohnson
2
370
Sky Analytics Pitch Deck
benbjohnson
0
590
Other Decks in Technology
See All in Technology
caching_sha2_passwordのはなし
boro1234
0
210
Cloud Native PG 使ってみて気づいたことと最新機能の紹介 - 第52回PostgreSQLアンカンファレンス
seinoyu
1
180
Engineering Managementのグローバルトレンド #emoasis / Engineering Management Global Trend
kyonmm
PRO
6
980
30代エンジニアが考える、エンジニア生存戦略~~セキュリティを添えて~~
masakiokuda
4
2k
Redefine_Possible
upsider_tech
0
240
バクラクでのSystem Risk Records導入による変化と改善の取り組み/Changes and Improvement Initiatives Resulting from the Implementation of System Risk Records
taddy_919
0
220
OCI見積もり入門セミナー
oracle4engineer
PRO
0
110
AI・LLM事業部のSREとタスクの自動運転
shinyorke
PRO
0
300
Cline、めっちゃ便利、お金が飛ぶ💸
iwamot
19
18k
ソフトウェアプロジェクトの成功率が上がらない原因-「社会価値を考える」ということ-
ytanaka5569
0
120
Explainable Software Engineering in the Public Sector
avandeursen
0
350
銀行でDevOpsを進める理由と実践例 / 20250317 Masaki Iwama
shift_evolve
1
100
Featured
See All Featured
Making the Leap to Tech Lead
cromwellryan
133
9.2k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Being A Developer After 40
akosma
90
590k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
31
4.7k
RailsConf 2023
tenderlove
29
1k
Fireside Chat
paigeccino
37
3.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.4k
YesSQL, Process and Tooling at Scale
rocio
172
14k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
The Cost Of JavaScript in 2023
addyosmani
48
7.6k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]