Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
分析AIエージェントを育てる、動く環境を育てる
Avatar for Codatum codatum
1
1k
Railsで挑む大量データ永続化 〜600億件を永続化するための設計とボトルネックとの戦い〜
Avatar for Shoichi Ochi occhi
0
160
【Data Engineering Summit2026_登壇資料】医療ビッグデータ×AIエージェントを支えるデータ基盤とハーネスエンジニアリング
Avatar for JMDC.Inc 登壇資料 jmdc_presentation
0
140
味覇で作るパスタが美味い - 20261001_yabaubuki.dev
Avatar for Livesense Inc. livesense
PRO
0
130
宣言型部分更新は何を解決しどこへ向かうのか
Avatar for did0es shuta13
1
360
継続的なAIコスト最適化
Avatar for Atsushi Tanaka bgpat
1
290
V8のArray.prototype.flatを最大約20倍速くするまでと、巨大OSSへの大規模コミットの道のり
Avatar for Riya Amemiya riyaamemiya
0
280
サブエージェントと Claude Mods で作る独自 Advisor ツールの話
Avatar for Kenichi Odo kenichiodo
1
120
Benchmarking Disaggregated Serving on vLLM + Dynamo with a Real Coding-Agent Workload
Avatar for NTT docomo Business nttcom
0
120
初対面のメンバーがチームを組み、Claude Codeでゼロからサービスをリリース
Avatar for k.masachika mkdev10
1
120
実体験から学ぶ分析エージェントの開発と運⽤
Avatar for Recruit recruitengineers
PRO
8
2.2k
Rustuna: Rustによる高速なOptuna実装
Avatar for Preferred Networks pfn
PRO
0
100

Featured

See All Featured
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1369
210k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
500
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
750
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
Avatar for Katarina Dahlin katarinadahlin
PRO
3
3.9k
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
5.3M
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
710
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
Avatar for Sara Fernández Carmona sarafernandez
0
300
Introduction to Domain-Driven Design and Collaborative software design
Avatar for Kenny Baas-Schwegler baasie
1
1k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
672
58k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.4k
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
300

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com