Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Databricks Free Editionで始めるLakeflow SDP
Avatar for Takaaki Yayoi taka_aki
0
120
I tried making a solo advent calendar!
Avatar for otsuka zzzzico
0
150
名刺メーカーDevグループ 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
1k
[PR] はじめてのデジタルアイデンティティという本を書きました
Avatar for ritou ritou
1
810
Oracle Database@Azure：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
3
380
複雑さを受け入れるか、拒むか？ - 事業成長とともに育ったモノリスを前に私が考えたこと #RSGT2026
Avatar for bayashi murabayashi
1
2k
これまでのネットワーク運用を変えるかもしれないアプデをおさらい
Avatar for da-hatakeyama hatahata021
3
200
習慣とAIと環境 — 技術探求を続ける3つの鍵
Avatar for azukiazusa azukiazusa1
2
530
善意の活動は、なぜ続かなくなるのか ーふりかえりが"構造を変える判断"になった半年間ー
Avatar for matsukurou matsukurou
0
550
The Engineer with a Three-Year Cycle
Avatar for YAMADA Nobuko e99h2121
0
150
みんなでAI上手ピーポーになろう！ / Let’s All Get AI-Savvy!
Avatar for 株式会社カミナシ kaminashi
0
150
あの夜、私たちは「人間」に戻った。 ── 災害ユートピア、贈与、そしてアジャイルの再構築 / 20260108 Hiromitsu Akiba
Avatar for SHIFT EVOLVE shift_evolve
PRO
0
710

Featured

See All Featured
What the history of the web can teach us about the future of AI
Avatar for Ines Montani inesmontani
PRO
0
400
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
72
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
34k
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
0
480
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
0
150
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
2
300
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
0
1.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
122
21k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
274
41k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com