Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
370
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
再考 アクターモデル/ reconsider actor model
ytake
0
380
「自動テストのプラクティスを効果的に学ぶためのカードゲーム」 ( #sqip2024 )
teyamagu
PRO
2
190
20240912 JJUGナイトセミナー
mii1004
0
140
たった1人からはじめる【Agile Community of Practice】~ソース原理とFearless Changeを添えて~
ktc_corporate_it
1
510
効果的なオンコール対応と障害対応
ryuichi1208
6
3.1k
開発生産性を始める前に開発チームができること / optim-improve-development-productivity.pdf
optim
0
150
エムスリーエビデンス創出プロダクトチーム紹介資料 / Introduction of M3 Create Evidence Team
m3_engineering
0
110
DevRelの始め方
moongift
PRO
2
400
Fediverse Discovery Providers overview
andypiper
0
170
持続可能なソフトウェア開発を支える『GitHub CI/CD実践ガイド』
tmknom
8
1.5k
「家族アルバム みてね」における運用管理・ オブザーバビリティの全貌 / Overview of Operation Management and Observability in FamilyAlbum
isaoshimizu
4
170
Analytics-Backed App Widget Development - Served with Jetpack Glance
miyabigouji
0
630
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
78
8.6k
Web Components: a chance to create the future
zenorocha
309
42k
Building a Scalable Design System with Sketch
lauravandoore
459
32k
Happy Clients
brianwarren
96
6.6k
A Tale of Four Properties
chriscoyier
155
22k
Automating Front-end Workflow
addyosmani
1365
200k
Making Projects Easy
brettharned
113
5.8k
Making the Leap to Tech Lead
cromwellryan
128
8.8k
Build The Right Thing And Hit Your Dates
maggiecrowley
30
2.3k
Practical Orchestrator
shlominoach
185
10k
Done Done
chrislema
180
16k
Agile that works and the tools we love
rasmusluckow
327
20k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com