Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
430
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
AIで政治は変わるのか？ — 中高生と考えたAI時代の民主主義（東海高校サタデープログラム）
Avatar for 須田英太郎 eitarosuda
0
390
インフラ寄りSREでも 開発に踏み出せる〜境界を越えてユーザー体験に向き合いたい〜
Avatar for SansanTech sansantech
PRO
0
700
グローバルチームと挑むプロダクト開発
Avatar for SansanTech sansantech
PRO
1
160
Text-to-SQLをAgentCoreで実現し、生成されるSQLの精度を定量的に評価する
Avatar for やくも yakumo
2
590
攻撃者がいなくてもAIエージェントはインシデントを起こす
Avatar for Nomizo Nomizo nomizone
0
200
Compose 新機能総まとめ / What's New in Jetpack Compose
Avatar for Yuki Anzai yanzm
0
130
ゼロをイチにする仕事が終わったあと
Avatar for Masato Sugiyama smasato
0
300
“全部コピーしない”ファイルデータの活用 ： — FSx for ONTAP × S3 Tables × Icebergで作るメタデータカタログ
Avatar for Yoshiki Fujiwara yoshiki0705
0
470
金融の未来を考える / Thinking About the Future of Finance
Avatar for Kenji Saito ks91
PRO
0
150
どうして今サーバーサイドKotlinを選択したのか
Avatar for Nealle nealle
0
200
AI Agent SaaS を支える自社仮想化基盤への挑戦と実運用 / ai-agent-saas-virtualization
Avatar for GMO Flatt Security flatt_security
2
1.3k
Zoom2Youtube.Claude
Avatar for Yasunobu Kawaguchi kawaguti
PRO
2
410

Featured

See All Featured
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
210
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.2k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
210k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
380
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
75
12k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.4k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
25k
Building AI with AI
Avatar for Ines Montani inesmontani
PRO
1
1.1k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.3k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
16k
KATA
Avatar for Megan Lloyd mclloyd
PRO
35
15k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
34k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com