$30 off During Our Annual Pro Sale. View Details »

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Reinforcement Fine-tuning 基礎〜実践まで
Avatar for Morita ch6noota
0
170
MLflowで始めるプロンプト管理、評価、最適化
Avatar for Databricks Japan databricksjapan
1
140
ログ管理の新たな可能性？CloudWatchの新機能をご紹介
Avatar for Ikumi Ono ikumi_ono
1
640
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
11
390k
AI駆動開発における設計思想 認知負荷を下げるフロントエンドアーキテクチャ/ 20251211 Teppei Hanai
Avatar for SHIFT EVOLVE shift_evolve
PRO
2
340
A Compass of Thought: Guiding the Future of Test Automation ( #jassttokai25 , #jassttokai )
Avatar for teyamagu teyamagu
PRO
1
250
Overture Maps Foundationの3年を振り返る
Avatar for Toru Mori moritoru
0
170
RAG/Agent開発のアップデートまとめ
Avatar for sakai taka0709
0
160
技術以外の世界に『越境』しエンジニアとして進化を遂げる 〜Kotlinへの愛とDevHRとしての挑戦を添えて〜
Avatar for subroh_0508 subroh0508
1
430
非CUDAの悲哀 〜Claude Code と挑んだ image to 3D “Hunyuan3D”を EVO-X2(Ryzen AI Max+395)で動作させるチャレンジ〜
Avatar for hawky the miscellaneous hawkymisc
1
170
AI活用によるPRレビュー改善の歩み ― 社内全体に広がる学びと実践
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
200
Snowflakeでデータ基盤を もう一度作り直すなら / rebuilding-data-platform-with-snowflake
Avatar for pei0804 pei0804
4
1.3k

Featured

See All Featured
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.3k
Docker and Python
Avatar for Tania Allard trallard
47
3.7k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
210
24k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.6k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
51k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
0
510
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
56
14k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
38
3k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
14k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com