Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
DI コンテナ自動生成ツールを実装してみた / intro-autodi
Avatar for uhzz uhzz
0
870
AI駆動開発でなんでもハンズオン環境をつくってみた
Avatar for yoshimi0227 yoshimi0227
0
160
大規模環境でどのように監視を実現する？
Avatar for Yuto Obayashi yuobayashi
1
150
A Harness for Behaviour: how to get AI to generate code that does what we intend, or "TDD in the age of AI"
Avatar for Matteo Vaccari xpmatteo
0
470
LLM時代のリファクタリング戦略_AIエージェントによる段階的・安全なTS移行方法
Avatar for PLAY, inc. play_inc
0
210
Sony_KMP_Journey_KotlinConf2026
Avatar for ソニー株式会社 sony
0
110
Kaggle未経験社員をメダリストに育てる「AIドラゴン桜」
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
630
oracle-to-databricks-migration-with-llm-and-dbt
Avatar for case-k-git casek
0
280
Claude Codeですべての日常業務を爆速化しよう！
Avatar for みのるん minorun365
PRO
16
14k
キャリア25年目にしてTypeScript に出会うまで - 「型」を通じて振り返るプログラミング言語遍歴 / Meeting TypeScript After 25 Years in Tech - Looking Back at My Programming Language Journey Through "Types"
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
2
290
Claude Code x Accounting
Avatar for Yasunobu Kawaguchi kawaguti
PRO
1
330
AI活用の格差をなくす：チーム全体のAI開発生産性を底上げする方法
Avatar for Atsushi Nakatsugawa moongift
PRO
1
110

Featured

See All Featured
It's Worth the Effort
Avatar for 3n 3n
188
29k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.9k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
200
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
240
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
6.9k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.2k
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.2k
Agile Leadership in an Agile Organization
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
150
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
540
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
273
27k
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
210

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com