Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
omasushiというライブラリを作った
Avatar for polidog polidog
PRO
0
150
GoにおけるFFIのこれまでとこれから
Avatar for Masaaki Goshima goccy
4
1.1k
Claude Code本って、 読む必要あるの？
Avatar for Oikon oikon48
1
130
目の前の楽しいが人生を変える - コミュニティの螺旋の歩き方と楽しむコツ / change your life
Avatar for soudai sone soudai
PRO
2
130
From Vanilla Kubernetes to a Batteries-Included Platform: Developer Experience at 1,300+ Clusters
Avatar for yosshi_ yosshi_
0
630
作り直せるコードは迅速に 作り直せないDBは慎重に - AI時代のプロダクトエンジニアが「判断の不可逆性」で開発速度を変える話
Avatar for kinosuke01 kinosuke01
0
280
日経電子版を支えていく Kasane Design System/fec_fukuoka
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
230
AIに丸投げしないトイル削減 / Eliminating Toil Without Leaving It All to AI
Avatar for kohbis kohbis
4
1k
Benchmarking Vector Databases: pgvector vs. LanceDB
Avatar for tsho tsho
0
150
AI時代に、プロダクトの数だけ積み上がる所有コストをどうエンジニアリングするか / Engineering the Cost of Ownership
Avatar for Kazuki Maeda kzkmaeda
1
1.1k
synctest時代のhttptest Go 1.27で変わるHTTPサーバテストの裏側 / go conference2026 synctest and httptest
Avatar for Yoichiro Shimizu budougumi0617
1
200
WAF 運用改善の承認サイクル/SRE_BizReach_MIXI_1
Avatar for Visional Engineering ＆ Design visional_engineering_and_design
2
370

Featured

See All Featured
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
430
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
2
560
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
Done Done
Avatar for chrislema chrislema
186
16k
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
820
Tips & Tricks on How to Get Your First Job In Tech
Avatar for Honza Javorek honzajavorek
1
730
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
2
410
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
580
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
187
23k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
230
23k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1033
470k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com