Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
430
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
AI研修(Day1)【MIXI 26新卒技術研修】
Avatar for MIXI ENGINEERS mixi_engineers
PRO
2
2.9k
AI Agent を本番環境へ―― Microsoft Foundry × Azure Serverless で作る Enterprise-Ready な基盤
Avatar for Tatsuro Shibamura shibayan
PRO
1
920
エンタープライズデータへ安全につなぐ Production-ready なエージェント設計 ― AI × MCP リファレンスアーキテクチャ ― #AIDevDay
Avatar for CData Software Japan cdataj
1
370
「待ち時間」の消滅と「自我消耗」の加速：生成AI時代のエンジニアを救うメンタル・リソース管理
Avatar for フクイ poropinai1966
0
370
QA・ソフトウェアテスト研修【MIXI 26新卒技術研修】
Avatar for MIXI ENGINEERS mixi_engineers
PRO
3
1.8k
13年運用タイトルのサーバーサイドが辿り着いた現在地 ― モンスターストライクにおける技術・組織・AI活用から得た知見
Avatar for MIXI ENGINEERS mixi_engineers
PRO
1
350
LLMリーダーボードアップデートに向けたAgentic Math_SWEのトレースについて
Avatar for YuyaYAMAMOTO nejumi
0
170
PLaMo 3.0 Primeの事後学習
Avatar for Preferred Networks pfn
PRO
0
210
なぜ、あなたのエージェントは言うことを聞かないのか
Avatar for segavvy segavvy
1
590
AI ネイティブな組織に Gemini Enterprise Agent Platform がなぜ必要なのか
Avatar for Asei Sugiyama asei
0
110
セキュリティ研修【MIXI 26新卒技術研修】
Avatar for MIXI ENGINEERS mixi_engineers
PRO
32
26k
GMOフィナンシャルゲートが挑む、「止まらない」決済インフラ構築の裏側【SORACOM Discovery 2026】
Avatar for SORACOM（ソラコム） soracom
PRO
0
110

Featured

See All Featured
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1.4k
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
400
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
Avatar for Tech SEO Connect techseoconnect
PRO
0
550
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.4k
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.2k
B2B Lead Gen: Tactics, Traps & Triumph
Avatar for Sophie Logan marketingsoph
0
190
HDC tutorial
Avatar for Michiel Stock michielstock
2
760
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
240
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
450
The Mindset for Success: Future Career Progression
Avatar for Greg Gifford greggifford
PRO
0
430

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com