Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
拝啓、あの夏の僕へ〜あなたも知っているApp Runnerの世界〜
Avatar for 荒木 ARK news_it_enj
0
200
ハーネスエンジニアリング入門
Avatar for 西岡 賢一郎 (Kenichiro Nishioka) knishioka
0
100
社内エンジニア勉強会の醍醐味と苦しみ/tamadev
Avatar for Ichiro Nishiuma nishiuma
0
280
COBOL婆さんの伝説
Avatar for フクイ poropinai1966
0
130
大学職員のための生成AI最前線 ：最前線を、AIガバナンスとして読み直すためのTips
Avatar for gmoriki | 森木銀河 gmoriki
1
3.2k
フロントエンドの相手が変わった - AIが加わったWebの新しいインターフェース設計
Avatar for azukiazusa azukiazusa1
30
9.3k
コードや知識を組み込む / Incorporate Code and Knowledge
Avatar for Kenji Saito ks91
PRO
0
200
【技術書典20】OpenFOAM（自宅で深める流体解析）流れと熱移動（２）
Avatar for kamakiri1225 kamakiri1225
0
360
AIと乗り切った1,500ページ超のヘルプサイト基盤刷新とさらにその先の話
Avatar for mugi / Hajime Mugishima mugi_uno
1
290
Shipping AI Agents — Lessons from Production
Avatar for vvatanabe vvatanabe
0
310
Modernizing Your HCL Connections Experience: Visual Report to chain, Profile Enhancements, and AI Integration
Avatar for Wannes Rams wannesrams
0
260
AIが盛んな時代に 技術記事を書き始めて起きた私の中での小さな変化
Avatar for 松尾淳平 peintangos
0
340

Featured

See All Featured
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
410
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
Avatar for Umar Saidu Auna auna
0
130
Public Speaking Without Barfing On Your Shoes - THAT 2023
Avatar for David Neal reverentgeek
1
380
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
122
21k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
180
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
220
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
2.7k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
2
1k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
672
58k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.6k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com