Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
AI駆動開発を実現するためのアーキテクチャと取り組み
Avatar for Yuichiro Yamashita baseballyama
2
910
個人から巡るAI疲れと組織としてできること - AI疲れをふっとばせ。エンジニアのAI疲れ治療法 ショートセッション -
Avatar for kichion kikuchikakeru
4
1.7k
なぜブラウザで帳票を生成したいのか　どのようにブラウザで帳票を生成するのか
Avatar for yagisan-reports yagisanreports
0
140
アジャイル社内普及ご近所さんマップを作ろう / Let's create an agile neighborhood map
Avatar for Satoshi Harada psj59129
1
140
Building AI Applications with Java, LLMs, and Spring AI
Avatar for Thomas Vitale thomasvitale
1
180
JJUG CCC 2025 Fall バッチ性能!!劇的ビフォーアフター
Avatar for HayashiYuichiro hayashiyuu1
1
370
大規模モノレポの秩序管理 失速しない多言語化フロントエンドの運用 / JSConf JP 2025
Avatar for shoota shoota
0
260
「O(n log(n))のパフォーマンス」の意味がわかるようになろう
Avatar for dhirabayashi dhirabayashi
0
190
ABEJA FIRST GUIDE for Software Engineers
Avatar for ABEJA abeja
0
3.2k
ソフトウェア開発現代史: 55％が変化に備えていない現実 ─ AI支援型開発時代のReboot Japan #agilejapan
Avatar for Hiroyuki TAKAHASHI takabow
7
4.4k
仕様は“書く”より“語る” - 分断を超えたチーム開発の実践 / 20251115 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
1.1k
プロジェクトの空気を読んで開発してくれるPerlのAIツールがほしい
Avatar for Kenta Kobayashi kfly8
2
100

Featured

See All Featured
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
280
24k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
1
30
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
33k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.3k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.5k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
29
5.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.6k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com