Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
370
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
OCI Vault 概要
oracle4engineer
PRO
0
9.7k
個人でもIAM Identity Centerを使おう!(アクセス管理編)
ryder472
4
230
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
160
【Pycon mini 東海 2024】Google Colaboratoryで試すVLM
kazuhitotakahashi
2
540
rootlessコンテナのすゝめ - 研究室サーバーでもできる安全なコンテナ管理
kitsuya0828
3
390
New Relicを活用したSREの最初のステップ / NRUG OKINAWA VOL.3
isaoshimizu
3
630
Platform Engineering for Software Developers and Architects
syntasso
1
520
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
8
870
Zennのパフォーマンスモニタリングでやっていること
ryosukeigarashi
0
150
Shopifyアプリ開発における Shopifyの機能活用
sonatard
4
250
強いチームと開発生産性
onk
PRO
35
11k
Why App Signing Matters for Your Android Apps - Android Bangkok Conference 2024
akexorcist
0
130
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Embracing the Ebb and Flow
colly
84
4.5k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
28
2k
Faster Mobile Websites
deanohume
305
30k
For a Future-Friendly Web
brad_frost
175
9.4k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
Speed Design
sergeychernyshev
25
620
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
329
21k
Practical Orchestrator
shlominoach
186
10k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com