Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beating State-of-the-art By -10000% @ CIDR Gong...
Search
Reynold Xin
January 07, 2013
Research
1
140
Beating State-of-the-art By -10000% @ CIDR Gong Show
I gave a 5-min Gong Show talk at CIDR on my experience with Spark, Shark, and GraphX.
Reynold Xin
January 07, 2013
Tweet
Share
More Decks by Reynold Xin
See All by Reynold Xin
(Berkeley CS186 guest lecture) Big Data Analytics Systems: What Goes Around Comes Around
rxin
12
2k
Interface Design for Spark Community
rxin
12
1.4k
Spark Committer Night meetup @ NYC
rxin
1
120
Apache Spark: Unified Platform for Big Data
rxin
1
230
Advanced Spark @ Spark Summit 2014
rxin
4
320
Apache Spark: Easier and Faster Big Data
rxin
2
280
GraphX at Spark User Meetup
rxin
0
140
Shark SIGMOD research deck
rxin
2
510
The Spark Ecosystem: Fast and Expressive Big Data Analytics in Scala @ Scala Days 2013
rxin
3
700
Other Decks in Research
See All in Research
Streamlit 総合解説 ~ PythonistaのためのWebアプリ開発 ~
mickey_kubo
2
1.4k
SSII2025 [TS3] 医工連携における画像情報学研究
ssii
PRO
2
1.3k
論文読み会 SNLP2025 Learning Dynamics of LLM Finetuning. In: ICLR 2025
s_mizuki_nlp
0
130
数理最適化と機械学習の融合
mickey_kubo
16
9.2k
電力システム最適化入門
mickey_kubo
1
870
When Submarine Cables Go Dark: Examining the Web Services Resilience Amid Global Internet Disruptions
irvin
0
290
A multimodal data fusion model for accurate and interpretable urban land use mapping with uncertainty analysis
satai
3
270
日本語新聞記事を用いた大規模言語モデルの暗記定量化 / LLMC2025
upura
0
140
まずはここから:Overleaf共同執筆・CopilotでAIコーディング入門・Codespacesで独立環境
matsui_528
2
410
AIによる画像認識技術の進化 -25年の技術変遷を振り返る-
hf149
7
3.9k
EarthSynth: Generating Informative Earth Observation with Diffusion Models
satai
3
190
Trust No Bot? Forging Confidence in AI for Software Engineering
tomzimmermann
1
260
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
338
57k
Being A Developer After 40
akosma
90
590k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
A designer walks into a library…
pauljervisheath
207
24k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Fireside Chat
paigeccino
39
3.6k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.5k
Embracing the Ebb and Flow
colly
87
4.8k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
1k
Transcript
Beating State-of-the-art By -10000% Reynold Xin, AMPLab, UC Berkeley with
help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
Beating State-of-the-art By -10000% NOT A TYPO Reynold Xin, AMPLab,
UC Berkeley with help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
MapReduce deterministic, idempotent tasks fault-tolerance elasticity resource sharing
“The bar for open source software is at historical low.”
“The bar for open source software is at historical low.”
i.e. “This is the right time to do grad school.”
iterative machine learning OLAP strong temporal locality
Does in-memory computation help in petabyte-scale warehouses?
Does in-memory computation help in petabyte-scale warehouses? YES
Spark How to do in-memory computation efficiently in a fault-tolerant
way?
Shark How to do SQL query processing efficiently in “MapReduce”
style SQL on top of Spark Hive compatible (UDF, Type, InputFormat, Metadata)
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.”
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.” i.e. “You should’ve come to grad school 2 years earlier.”
Shark in-memory columnar store dynamic query re-optimization and a lot
of engineering...
Query 1 Query 2 Log Regress 0 20 40 60
80 100 120 110 94 64 0.96 1 0.7 Runtime (seconds) on a 100-node EC2 cluster Shark/Spark Hive/Hadoop
iterative machine learning SQL query processing
iterative machine learning SQL query processing graph computation
GraphLab on Spark
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. Not bad for a day of work!
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. but I later found out that it is still 10X slower than the latest version of GraphLab :(
A lot of open questions for fault- tolerant, distributed graph
computation. “MapReduce”? Data partitioning? Fault-tolerance? Asynchrony?
iterative machine learning www.spark-project.org SQL query processing shark.cs.berkeley.edu graph computation
www.wait-another-year.com