Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
作って終わりにしない タイミーのセマンティックレイヤー育成の現在地
Avatar for chanyou0311 chanyou0311
4
2.3k
2026TECHFRESH畢業分享會 - Lightning Talk - 資料也要 CI/CD? 用 Airbyte 自動化資料同步
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
930
Disciplined Vibes: Scaling AI-Assisted Engineering
Avatar for Sheharyar Naseer sheharyar
0
140
2026TECHFRESH畢業分享會 - Lightning Talk - E起 See See : 電商推薦讀心術? 數據說了算
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
940
2026 TECHFRESH 畢業分享會 - 開發日常大解密！從領域驅動到企業級上線
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
940
自律型AIエージェントは何を破壊するのか
Avatar for kojira kojira
0
160
AIはどのように 組織のアジリティを変えるのか？
Avatar for J.K junki
2
670
プロダクト開発から業務改善コンサルまで。事業全体へ「染み出す」ことで広がるエンジニアの可能性
Avatar for ham ham0215
0
120
SONiCのLinuxベースを活かしたZabbix監視
Avatar for SONiC Users Group Japan sonic
0
110
NAB Show 2026 動画技術関連レポート / NAB Show 2026 Report
Avatar for CyberAgent cyberagentdevelopers
PRO
0
190
FDE という解 ― 暗黙知と明示知をつなぐ、伴走型エンジニアリング ―
Avatar for 太田 博三 otanet
0
150
失敗を資産に変えるClaude Code
Avatar for Shinya Saita shinyasaita
0
620

Featured

See All Featured
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.3k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
34k
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.3k
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
310
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
210k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
25k
Ethics towards AI in product and experience design
Avatar for Skipper Chong Warson skipperchong
2
310
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
75
12k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
210
Lightning Talk: Beautiful Slides for Beginners
Avatar for Ines Montani inesmontani
PRO
2
570
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
720
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
55
12k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com