Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
410
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
210
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
AIペネトレーションテスト・ セキュリティ検証「AgenticSec」ご紹介資料
Avatar for Sho Nakatani laysakura
0
2.1k
Introduction to Bill One Development Engineer
Avatar for Sansan, Inc. sansan33
PRO
0
410
AgentCore RuntimeからS3 Filesをマウントしてみる
Avatar for Har1101 har1101
4
430
サイバーフィジカル社会とは何か / What Is a Cyber-Physical Society?
Avatar for Kenji Saito ks91
PRO
0
180
Zero Data Loss Autonomous Recovery Service サービス概要
Avatar for oracle4engineer oracle4engineer
PRO
5
14k
Contract One Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
16k
ルールルルルル私的函館観光ガイド── 函館の街はイクラでも楽しめる！
Avatar for nomuson nomuson
0
190
最近の技術系の話題で気になったもの色々（IoT系以外も） / IoTLT 花見予定会（たぶんBBQ） @都立潮風公園バーベキュー広場
Avatar for you（@youtoy） you
PRO
1
140
暗黙知について一歩踏み込んで考える - 暗黙知の４タイプと暗黙考・暗黙動へ
Avatar for Masaya Mori (森正弥) / CAIO (Chief AI Officer) masayamoriofficial
0
1.5k
ワールドカフェI ／チューターを改良する / World Café I and Improving the Tutors
Avatar for Kenji Saito ks91
PRO
0
220
Master Dataグループ紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
4.6k
新規サービス開発におけるReact Nativeのリアル〜技術選定の裏側と実践的OSS活用〜
Avatar for Takahiro Kato grandbig
2
190

Featured

See All Featured
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.5k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
Avatar for Greg Gifford greggifford
PRO
0
140
Lightning Talk: Beautiful Slides for Beginners
Avatar for Ines Montani inesmontani
PRO
1
510
My Coaching Mixtape
Avatar for mlcsv mlcsv
0
97
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
160
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
Avatar for Tomoya Matsuura tomoyanonymous
2
690
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
61
43k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
Avatar for Aleyda Solis aleyda
1
1.9k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
122
21k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
810
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
4
290

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com