Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beating State-of-the-art By -10000% @ CIDR Gong...
Search
Reynold Xin
January 07, 2013
Research
1
140
Beating State-of-the-art By -10000% @ CIDR Gong Show
I gave a 5-min Gong Show talk at CIDR on my experience with Spark, Shark, and GraphX.
Reynold Xin
January 07, 2013
Tweet
Share
More Decks by Reynold Xin
See All by Reynold Xin
(Berkeley CS186 guest lecture) Big Data Analytics Systems: What Goes Around Comes Around
rxin
12
2k
Interface Design for Spark Community
rxin
12
1.4k
Spark Committer Night meetup @ NYC
rxin
1
120
Apache Spark: Unified Platform for Big Data
rxin
1
230
Advanced Spark @ Spark Summit 2014
rxin
4
330
Apache Spark: Easier and Faster Big Data
rxin
2
290
GraphX at Spark User Meetup
rxin
0
140
Shark SIGMOD research deck
rxin
2
520
The Spark Ecosystem: Fast and Expressive Big Data Analytics in Scala @ Scala Days 2013
rxin
3
700
Other Decks in Research
See All in Research
CoRL2025速報
rpc
1
1.9k
[RSJ25] Enhancing VLA Performance in Understanding and Executing Free-form Instructions via Visual Prompt-based Paraphrasing
keio_smilab
PRO
0
140
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
akihiro_kokubo
PRO
62
31k
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
tomoyanonymous
0
200
AI エージェントを活用した研究再現性の自動定量評価 / scisci2025
upura
1
160
心理言語学の視点から再考する言語モデルの学習過程
chemical_tree
2
610
最適化と機械学習による問題解決
mickey_kubo
0
180
とあるSREの博士「過程」 / A Certain SRE’s Ph.D. Journey
yuukit
11
4.4k
GPUを利用したStein Particle Filterによる点群6自由度モンテカルロSLAM
takuminakao
0
350
Google Agent Development Kit (ADK) 入門 🚀
mickey_kubo
2
2k
Towards a More Efficient Reasoning LLM: AIMO2 Solution Summary and Introduction to Fast-Math Models
analokmaus
2
890
MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation
satai
4
310
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
96
6.3k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
37
2.6k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Six Lessons from altMBA
skipperchong
28
4k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
For a Future-Friendly Web
brad_frost
180
9.9k
BBQ
matthewcrist
89
9.8k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
4 Signs Your Business is Dying
shpigford
185
22k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Code Reviewing Like a Champion
maltzj
525
40k
Building Applications with DynamoDB
mza
96
6.7k
Transcript
Beating State-of-the-art By -10000% Reynold Xin, AMPLab, UC Berkeley with
help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
Beating State-of-the-art By -10000% NOT A TYPO Reynold Xin, AMPLab,
UC Berkeley with help from Joseph Gonzalez, Josh Rosen, Matei Zaharia, Michael Franklin, Scott Shenker, Ion Stoica
MapReduce deterministic, idempotent tasks fault-tolerance elasticity resource sharing
“The bar for open source software is at historical low.”
“The bar for open source software is at historical low.”
i.e. “This is the right time to do grad school.”
iterative machine learning OLAP strong temporal locality
Does in-memory computation help in petabyte-scale warehouses?
Does in-memory computation help in petabyte-scale warehouses? YES
Spark How to do in-memory computation efficiently in a fault-tolerant
way?
Shark How to do SQL query processing efficiently in “MapReduce”
style SQL on top of Spark Hive compatible (UDF, Type, InputFormat, Metadata)
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.”
“You need to beat Hadoop by at least 100X to
publish a paper in 2013.” i.e. “You should’ve come to grad school 2 years earlier.”
Shark in-memory columnar store dynamic query re-optimization and a lot
of engineering...
Query 1 Query 2 Log Regress 0 20 40 60
80 100 120 110 94 64 0.96 1 0.7 Runtime (seconds) on a 100-node EC2 cluster Shark/Spark Hive/Hadoop
iterative machine learning SQL query processing
iterative machine learning SQL query processing graph computation
GraphLab on Spark
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. Not bad for a day of work!
I spent a day pair-programming with Joey Gonzalez and improved
performance by 10X. but I later found out that it is still 10X slower than the latest version of GraphLab :(
A lot of open questions for fault- tolerant, distributed graph
computation. “MapReduce”? Data partitioning? Fault-tolerance? Asynchrony?
iterative machine learning www.spark-project.org SQL query processing shark.cs.berkeley.edu graph computation
www.wait-another-year.com