Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
Strands Agents & Bedrock AgentCoreを1分でおさらい
Avatar for みのるん minorun365
PRO
6
240
Nx × AI によるモノレポ活用 〜コードジェネレーター編〜
Avatar for puku0x puku0x
0
360
alecthomas/kong はいいぞ
Avatar for FUJIWARA Shunichiro fujiwara3
6
1.4k
相互運用可能な学修歴クレデンシャルに向けた標準技術と国際動向
Avatar for Naohiro Fujie fujie
0
210
【CEDEC2025】『ウマ娘 プリティーダービー』における映像制作のさらなる高品質化へ！～ 豊富な素材出力と制作フローの改善を実現するツールについて～
Avatar for Cygames cygames
PRO
0
230
Kiroでインフラ要件定義～テスト
 を実施してみた
Avatar for nagisa_53 nagisa53
3
310
Amazon Bedrock AgentCoreのフロントエンドを探す旅 (Next.js編)
Avatar for たけのこ kmiya84377
1
120
「AIと一緒にやる」が当たり前になるまでの奮闘記
Avatar for KAKEHASHI kakehashi
PRO
3
100
みんなのSRE 〜チーム全員でのSRE活動にするための4つの取り組み〜
Avatar for KAKEHASHI kakehashi
PRO
2
140
金融サービスにおける高速な価値提供とAIの役割 #BetAIDay
Avatar for LayerX layerx
PRO
1
750
Mambaで物体検出 完全に理解した
Avatar for Reiki Shirasawa shirarei24
2
220
【Λ（らむだ）】最近のアプデ情報 / RPALT20250729
Avatar for Λ lambda
0
230

Featured

See All Featured
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
207
24k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
8
750
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
1
530
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
614
210k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
Adopting Sorbet at Scale
Avatar for Ufuk Kayserilioglu ufuk
77
9.5k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
173
14k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
Faster Mobile Websites
Avatar for Dean Hume deanohume
308
31k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com