Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
人材育成分科会.pdf
Avatar for _awache _awache
2
160
あなたの知らないPDFのアクセシビリティ
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
160
LLMにもCAP定理があるという話
Avatar for Haruka Sakihara harukasakihara
0
330
失敗を経て、Harness Engineering で 大切にしたいことを考える / Learning from Failure: What Matters in Harness Engineering
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
1
350
プロダクト開発から業務改善コンサルまで。事業全体へ「染み出す」ことで広がるエンジニアの可能性
Avatar for ham ham0215
0
120
データサイエンスを価値につなげるプロジェクト設計 〜 DS一年目が現場で得た気づき 〜
Avatar for Yusei Doi ysd113
1
220
「エンジニア進化論」2028年の開発完全自動化、エンジニアはどう進化するか
Avatar for CyberAgent cyberagentdevelopers
PRO
6
5k
2026TECHFRESH畢業分享會 - AI 時代的人生存檔點
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
940
SONiCのLinuxベースを活かしたZabbix監視
Avatar for SONiC Users Group Japan sonic
0
110
2026TECHFRESH畢業分享會 - 葬送的通靈師：化系統與用戶雜訊成行動訊號
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
940
AIのReact習熟度を測る
Avatar for uhyo uhyo
2
350
Oracle AI Database@Azure：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
6
2k

Featured

See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.9k
BBQ
Avatar for Matthew Crist matthewcrist
89
10k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
66k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
200
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.4k
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
230
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
370
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
Avatar for Tech SEO Connect techseoconnect
PRO
0
180
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
250
1.3M
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
11
38k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
320

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com