Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
ファイルシステムの不整合
Avatar for Satoru Takeuchi sat
PRO
2
110
GitHub Copilot CLI 現状確認会議(2026年2月のすがた)
Avatar for Toru Makabe torumakabe
2
350
"共通化"と"Embed"のブレンドでスケール可能な運用を！M&Aを支えるGENDA SREの実践 / GENDA Tech Talk #3
Avatar for GENDA genda
0
200
Claude Code で画面の仕様書を作ろう
Avatar for ZOZO Developers zozotech
PRO
0
300
「データの価値を、みんなの武器に。」Data Enablementの価値とツラみ
Avatar for Ryosuke Noaki ryoskdara_
1
120
衛星画像即時マッピングサービスの実現に向けて
Avatar for Yuji Kobayashi lehupa
1
280
『誰の責任？』で揉めるのをやめて、エラーバジェットで判断するようにした ～感情論をデータで終わらせる、PMとエンジニアの意思決定プロセス～
Avatar for coconala_engineer coconala_engineer
0
860
私たち準委任PdEは2つのプロダクトに挑戦する ～ソフトウェア、開発支援という”二重”のプロダクトエンジニアリングの実践～ / 20260212 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
3
340
【Developers Summit 2026】Memory Is All You Need:コンテキストの「最適化」から「継続性」へ ～RAGを進化させるメモリエンジニアリングの最前線～
Avatar for KoheiOgawa shisyu_gaku
3
210
チーム開発の基礎_研究を事業につなげるために
Avatar for CyberAgent cyberagentdevelopers
PRO
7
3.5k
既存のログ監視システムをクラウドっぽく実装してみた
Avatar for tjmtrhs tjmtrhs
0
100
AIエージェントに必要なのはデータではなく文脈だった/ai-agent-context-graph-mybest
Avatar for Jun Naito jonnojun
1
660

Featured

See All Featured
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
180
Highjacked: Video Game Concept Design
Avatar for Ryan Kendrick rkendrick25
PRO
1
300
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
370
Google's AI Overviews - The New Search
Avatar for Barry Adams badams
0
920
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
Tips & Tricks on How to Get Your First Job In Tech
Avatar for Honza Javorek honzajavorek
0
440
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
75
5.1k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.2k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
1
60
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
66k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com