Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Eight Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
3
8.2k
医療の現場を変革に挑戦した半年間の軌跡 - PythonとAIで現場を変える / From Code to Care
Avatar for soudai sone soudai
PRO
0
390
三人寄ればチューリング完全
Avatar for Takumi Sueda puhitaku
4
1.8k
LanceDB入門
Avatar for Tomoko Uchida mocobeta
9
640
NANDでも描画したい！
Avatar for nichica nichica906
0
260
AI for Science時代を切り開く、政府の次世代HPC戦略の展望
Avatar for GPU UNITE gpuunite_official
0
130
Android skills に学ぶ
Avatar for kokiko kokiko
0
130
FORENSIA: ローカルLLMフォレンジックハーネス
Avatar for S.Nakano sumeshi
2
430
[potatotips #96] Give Your AI Agent the Flutter Playbook
Avatar for Kenichi Kambara korodroid
0
120
Head First モブプログラミング / Head First Mobprogramming
Avatar for TAKAKING22 takaking22
10
12k
AIのためのEthernet技術動向 (SerDes)
Avatar for Masayuki Kobayashi markunet
1
260
DatadogのBits Chatが開発組織にもたらしたもの / What Bits Chat Has Brought Us
Avatar for SMS tech sms_tech
1
290

Featured

See All Featured
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
73
41k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
190
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
96
14k
Accessibility Awareness
Avatar for Sarah Abderemane sabderemane
1
180
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.3k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
1
2.4k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
11k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
500
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2.1k
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.8k
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
780

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com