Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
210
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
親子 or ペアで Mashup for the Future！ しゃべって楽しむ 初手AI駆動でものづくり体験
Avatar for hiroramos hiroramos4
PRO
0
100
[JAWS DAYS 2026]私の AWS DevOps Agent 推しポイント
Avatar for Toshihiro Furuno furuton
0
140
JAWS DAYS 2026 楽しく学ぼう！ストレージ 入門
Avatar for Yoshiki Fujiwara yoshiki0705
2
140
OCI技術資料 : コンピュート・サービス 概要
Avatar for Oracle Cloud Infrastructure ソリューション・エンジニア ocise
4
54k
非情報系研究者へ送る Transformer入門
Avatar for Ryo Ishiyama rishiyama
11
7k
ナレッジワークのご紹介（第88回情報処理学会 ）
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
0
180
20260311 ビジネスSWG活動報告（デジタルアイデンティティ人材育成推進WG Ph2 活動報告会）
Avatar for OpenID Foundation Japan oidfj
0
250
20260311 技術SWG活動報告（デジタルアイデンティティ人材育成推進WG Ph2 活動報告会）
Avatar for OpenID Foundation Japan oidfj
0
280
プロジェクトマネジメントをチームに宿す -ゼロからはじめるチームプロジェクトマネジメントは活動1年未満のチームの教科書です- / 20260304 Shigeki Morizane
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
250
Claude Code Skills 勉強会 (DevelersIO向けに調整済み) / claude code skills for devio
Avatar for MasahiroKawahara masahirokawahara
1
14k
マネージャー版 "提案のレベル" を上げる
Avatar for konifar konifar
22
15k
Kubernetesにおける推論基盤
Avatar for ry ry
1
300

Featured

See All Featured
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6k
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
1
2k
Navigating the moral maze — ethical principles for Al-driven product design
Avatar for Skipper Chong Warson skipperchong
2
280
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
3k
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.4k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
2.8k
Between Models and Reality
Avatar for mayunak mayunak
2
230
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
14k
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
150
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
940
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
9
36k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com