Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Siddhartha Reddy
May 15, 2015
Technology
0
380

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k

Other Decks in Technology

See All in Technology
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
14
3.6k
オブザーバビリティの観点でみるAWS / AWS from observability perspective
ymotongpoo
8
1.5k
リアルタイム分析データベースで実現する SQLベースのオブザーバビリティ
mikimatsumoto
0
1.4k
エンジニアが加速させるプロダクトディスカバリー 〜最速で価値ある機能を見つける方法〜 / product discovery accelerated by engineers
rince
4
380
Classmethod AI Talks(CATs) #17 司会進行スライド(2025.02.19) / classmethod-ai-talks-aka-cats_moderator-slides_vol17_2025-02-19
shinyaa31
0
130
ビジネスモデリング道場　目的と背景
masuda220
PRO
9
560
あれは良かった、あれは苦労したB2B2C型SaaSの新規開発におけるCloud Spanner
hirohito1108
2
640
利用終了したドメイン名の最強終活〜観測環境を育てて、分析・供養している件〜 / The Ultimate End-of-Life Preparation for Discontinued Domain Names
nttcom
2
200
明日からできる！技術的負債の返済を加速するための実践ガイド～『ホットペッパービューティー』の事例をもとに～
recruitengineers
PRO
3
410
Oracle Cloud Infrastructure：2025年2月度サービス・アップデート
oracle4engineer
PRO
1
220
RSNA2024振り返り
nanachi
0
590
開発組織のための セキュアコーディング研修の始め方
flatt_security
3
2.4k

Featured

See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
Producing Creativity
orderedlist
PRO
344
39k
Six Lessons from altMBA
skipperchong
27
3.6k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
630
Designing Experiences People Love
moore
140
23k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.1k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
27
1.9k
Building an army of robots
kneath
303
45k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com