Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Oracle Cloud Infrastructure データベース・クラウド：各バージョンのサポート期間
Avatar for oracle4engineer oracle4engineer
PRO
57
47k
登壇駆動学習のすすめ — CfPのネタの見つけ方と書くときに意識していること
Avatar for おおいし bicstone
3
340
SREチームをどう作り、どう育てるか ― Findy横断SREのマネジメント
Avatar for adachi.ryo rvirus0817
0
500
猫でもわかるKiro CLI(セキュリティ編)
Avatar for Hiroo Katoh kentapapa
1
250
pool.ntp.orgに ⾃宅サーバーで 参加してみたら...
Avatar for Fuminori -Tany- Tanizaki(谷崎文義) tanyorg
1
3k
Amazon Rekognitionで 「信玄餅きなこ問題」を解決する
Avatar for usanchuu usanchuu
1
390
マネージャー視点で考えるプロダクトエンジニアの評価 / Evaluating Product Engineers from a Manager's Perspective
Avatar for hiro-torii hiro_torii
0
330
今こそ学びたいKubernetesネットワーク ～CNIが繋ぐNWとプラットフォームの「フラッと」な対話
Avatar for Takuto Nagami logica0419
9
760
22nd ACRi Webinar - NTT Kawahara-san's slide
Avatar for Nao Sumikawa nao_sumikawa
0
140
AIエージェントに必要なのはデータではなく文脈だった/ai-agent-context-graph-mybest
Avatar for Jun Naito jonnojun
1
660
GitHub Copilot CLI 現状確認会議(2026年2月のすがた)
Avatar for Toru Makabe torumakabe
2
360
2026年のAIエージェント構築はどうなる？
Avatar for みのるん minorun365
7
1.2k

Featured

See All Featured
Accessibility Awareness
Avatar for Sarah Abderemane sabderemane
0
65
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.6k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.4k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
2.5k
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
From π to Pie charts
Avatar for Rasagy Sharma rasagy
0
140
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
460
Are puppies a ranking factor?
Avatar for Jono Alderson jonoalderson
1
3k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.4k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
120

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com