Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
360
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
The Knowledge Spine: A Machine-Executable Ontology for Governed Marketing Activation
Avatar for Ananth Packkildurai vananth22
0
110
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
160
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
98
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
650
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
450
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
260
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k

Other Decks in Programming

See All in Programming
Heart of Swift Concurrency
Avatar for Yuta Koshizawa koher
0
890
Snowflakeで業務アプリを作ろう。 Snowflakeのアプリ機能解説&実践ガイド
Avatar for Ayumu Yamaguchi ayumu_yamaguchi
1
280
Go × SIMDで高速化するベクトル検索 ~ルーフラインモデルでSIMDが効く境界を探れ！ ~
Avatar for po3rin po3rin
1
1.7k
wkhtmltopdfの次どうするか問題2026
Avatar for Shinichi Maeshima willnet
2
1.4k
すこし踏み込む CancellationToken
Avatar for tkym htkym
1
480
テストを司るデーモンに会いに行く 〜隔離した仮想マシンでテストを通すまで〜
Avatar for Hidemune Takahashi h1d3mun3
1
440
フロントエンドUIフレームワークのこれまでとこれから
Avatar for TOMIKAWA Sotaro ssssota
5
2.8k
WebAssembly in Android Apps 〜 WASMはJNIの夢を見るか
Avatar for ARIYAMA Keiji keiji
1
110
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
440
スマート反転とウェブアクセシビリティ
Avatar for KAMIHARA Takumi camiha
0
210
変化を抱擁するドキュメントの作り方 - ビジネスルール駆動開発がもたらす、コードとの新しい関係
Avatar for ioki ioki
2
210
App Storeの外へ──日本のiOSサイドローディング入門 for iOSDC Japan 2026
Avatar for 野瀬田　裕樹 yuukiw00w
0
220

Featured

See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.4k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
1
360
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
360
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
330
AI in Enterprises - Java and Open Source to the Rescue
Avatar for ivargrimstad ivargrimstad
0
1.5k
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
3.1k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
590
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.4k
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.9k
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
490
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.6k
State of Search Keynote: SEO is Dead Long Live SEO
Avatar for Ryan Jones ryanjones
0
280

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!