Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
140
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
Avatar for 弁護士ドットコム bengo4com
0
1.3k
Oracle Cloud Infrastructure：2025年6月度サービス・アップデート
Avatar for oracle4engineer oracle4engineer
PRO
2
310
解析の定理証明実践＠Lean 4
Avatar for H Segawa dec9ue
1
200
FOSS4G 2025 KANSAI　QGISで点群データをいろいろしてみた
Avatar for kou_kita kou_kita
0
240
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
150
KubeCon + CloudNativeCon Japan 2025 Recap Opening & Choose Your Own Adventureシリーズまとめ
Avatar for mm-matsuda mmmatsuda
0
230
PHPでWebブラウザのレンダリングエンジンを実装する
Avatar for ディップ株式会社 dip_tech
PRO
0
220
Tokyo_reInforce_2025_recap_iam_access_analyzer
Avatar for h-ashisan hiashisan
0
140
「良さそう」と「とても良い」の間には 「良さそうだがホンマか」がたくさんある / 2025.07.01 LLM品質Night
Avatar for Shumpei Miyawaki smiyawaki0820
1
430
Lazy application authentication with Tailscale
Avatar for Elliot Blackburn bluehatbrit
0
110
Tech-Verse 2025 Keynote
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
1.3k

Featured

See All Featured
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.8k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
329
21k
The Language of Interfaces
Avatar for Des Traynor destraynor
158
25k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
54
6.4k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.7k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
94
14k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
16k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.7k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.6k
Scaling GitHub
Avatar for Zach Holman holman
459
140k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com