Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
370
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
現地でMeet Upをやる場合の注意点〜反省点を添えて〜
shotashiratori
0
530
大規模データ基盤チームのオンプレTiDB運用への挑戦 / dpu-tidb
cyberagentdevelopers
PRO
1
110
事業者間調整の行間を読む 調整の具体事例
sugiim
0
1.5k
Amazon_CloudWatch_ログ異常検出_導入ガイド
tsujiba
4
1.6k
小規模に始めるデータメッシュとデータガバナンスの実践
kimujun
3
590
Amazon FSx for NetApp ONTAPを利用するにあたっての要件整理と設計のポイント
non97
1
160
MAMを軸とした動画ハンドリングにおけるAI活用前提の整備と次世代ビジョン / abema-ai-mam
cyberagentdevelopers
PRO
1
120
[JAWS-UG金沢支部×コンテナ支部合同企画]コンテナとは何か
furuton
3
260
よくわからんサービスについての問い合わせが来たときの強い味方 Amazon Q について
kazzpapa3
0
220
Commitment vs Harrisonism - Keynote for Scrum Niseko 2024
miholovesq
6
1.1k
君は隠しイベントを見つけれるか?
mujyun
0
290
生成AIと知識グラフの相互利用に基づく文書解析
koujikozaki
1
140
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
327
21k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
14
1.9k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
167
49k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2.1k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
Optimising Largest Contentful Paint
csswizardry
33
2.9k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
[RailsConf 2023] Rails as a piece of cake
palkan
51
4.9k
A Philosophy of Restraint
colly
203
16k
GraphQLの誤解/rethinking-graphql
sonatard
66
9.9k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com