Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Etsy on Migrating to Kafka (in three short years)
Search
Hakka Labs
January 22, 2015
Programming
6.1k
4
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Etsy on Migrating to Kafka (in three short years)
Full post with video here:
Hakka Labs
January 22, 2015
Transcript
68 slides
Migrating to Kafka in Three Short Years A look at
the choices that defined the Etsy analytics stack
None
Path Dependence
Decisions made in the past limit options in the present,
even if the circumstances under which those past decisions were made are no longer relevant.
In other words, we can’t upgrade the Hadoop cluster until
we port all of the Cascading.jruby jobs to Scalding.
Sneak Preview ! 1. How Etsy built its original analytics
stack 2. Handling changes prepared us to rebuild our data pipeline 3. Kafka!
Starting from scratch
Choice #1 ! Acquire Adtuitive
None
None
Before you can work on search, you need real analytics
Choice #2 ! Build a zero-impact analytics stack
Etsy is not a cloud company but the first analytics
stack was cloud-based
(illustration here) browser CDN EMR S3 mysql FTP
Legacy effects: ! 24 hour latency on events 48 hour
latency on visits
Choice #3 ! Cascading.jruby
Hadoop Cascading Cascading.jruby
Choice #4 ! Use GA _utma cookie to define visits
Benefits: ! •Simpler ETL •Visits computed on the client side
•Easy to reconcile against Google Analytics
Choice #5 ! Using existing feature library for A/B tests
Leveraged existing experience with operational ramp-ups
Low impact: just required a logging change
Choice #6 ! Build analytics stack around visit-level metrics
Great for search and ads, less great for measuring engagement
Changing the tires without stopping the car
How do we instrument the iOS app? Summer 2012
1. Native app visits should have the same structure as
Web visits
2. Native app events should use the existing data pipeline
3. The native app should buffer events and send them
when convenient
Solution: ! 1. App uploads bundles of events to API
endpoint 2. Backend event logger curls the beacon for every event
Side effect: ! We have a backend event logger that
is now used all over the place
CDN diversification project Fall 2012
None
Migrated to our own beacon infrastructure
Data pipeline based on Apache, PHP, logrotate, and cron
We built our own Hadoop cluster: Etsydoop Fall 2012
We hired the Scalding guy Fall 2012
Hadoop Cascading Cascading.jruby Scalding
None
Uh oh, the Google Analytics JS hurts performance Fall 2012
The event logger’s GA dependency precluded async loading, hurting performance
First idea: duplicate the _utma functionality in our own code
The trouble with backend events
Visit Time Logger Event Type 1 12:01 frontend home 1
12:03 backend login 1 12:03 frontend view listing 1 1:31 backend logout 2 1:31 frontend view listing 2 1:32 frontend search 2 1:33 frontend view listing wrong visit
Complete rewrite of our ETL jobs Spring/Summer 2013
None
Backend page-view events Fall 2013
None
2014: the next phase
EventPipe goals
Use POST rather than multiple GET requests to prevent data
loss
Use JSON rather than query strings for comprehensibility
Validate beacon data before it enters the data pipeline
Use a binary serialization format for long-term storage
Use Kafka for data transfer to escape the batch paradigm
Eliminate individual beacon servers as points of failure
How do we handle the impedance mismatch between Apache/PHP and
Kafka?
Wrote a server in Go to serialize beacons in Thrift
and send them to Kafka
Use Apache for SSL termination
Still to come
Real-ish time ETL
Streaming infrastructure
Offline processing for more products
Other Kafka applications
Takeaways
Every choice you make has long-term implications
Fixing stuff creates new opportunities
@rafeco http://rc3.org
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.9k
Groupcache with Evan Owen
hakka_labs
2
5.4k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2.1k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5.1k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
エラーチャンネルのボールを拾い続けて鍛えるエンジニアの地力 / phpcon-niigata2026
meihei3
1
470
Proxmox x Talos Linux x Terraform で実現するなんかいい感じの Kubernetes クラスタ / A Sleek Kubernetes Cluster Powered by Proxmox, Talos Linux, and Terraform
nnaka2992
0
270
Coding with AI: How to trust the storyteller (AWS Community Day Adria, Ljubljana 2026)
slobodan
0
150
将来の標準フォーマッター?ktfmtを深堀る
t45k
0
220
AI駆動開発4年目の現在地(『あなたはFDEになるべきか?』イベント資料)
kumaiu
0
670
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか?〜 / Introduction to IaC in the AI Era
akihisaikeda
6
1.6k
なぜ私たちは「現時点では」KMPの導入を見送ったのか
chigichan24
1
310
[DroidKnights 2026] Workflow AX 적용기 — 자동 정책 정리부터 E2E까지
dayeon
0
120
オーケストレーション記述言語 Katari
yukikurage
0
270
protojson における文字化けの扱い
kazuki1023
0
450
はじめてのPLATEAU
nokonoko1203
1
660
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
2
380
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
38
3.1k
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
aleyda
1
2.3k
Intergalactic Javascript Robots from Outer Space
tanoku
273
28k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
1.1k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
18k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
56
3.5k
Context Engineering - Making Every Token Count
addyosmani
9
1.3k
Statistics for Hackers
jakevdp
799
230k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
uxyall
2
540
A brief & incomplete history of UX Design for the World Wide Web: 1989–2019
jct
2
570
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
techseoconnect
PRO
0
760
Docker and Python
trallard
47
4.3k