Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Deploy your own Spark cluster in 4 minutes usin...
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
Pishen Tsai
December 05, 2015
Programming
650
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Deploy your own Spark cluster in 4 minutes using sbt.
Pishen Tsai
December 05, 2015
Transcript
25 slides
Pishen Tsai @ KKBOX Deploy your own Spark cluster in
4 minutes using sbt
KKBOX / spark-deployer • SBT plugin. • Productively used in
KKBOX. • 100% Scala. https://github.com/KKBOX/spark-deployer
None
destroy cluster submit job create cluster write the code compile
& assembly
• spark-ec2 • amazon emr (Elastic MapReduce) • spark-deployer Solutions
https://aws.amazon.com/elasticmapreduce/details/spark http://spark.apache.org/docs/latest/ec2-scripts.html spark-ec2: amazon emr:
• spark-ec2 • amazon emr (Elastic MapReduce) • spark-deployer Solutions
spark-ec2 write the code compile & assembly submit job create
cluster destroy cluster sbt scp & ssh spark-ec2 spark-ec2
spark-ec2’s commands $ sbt assembly $ spark-ec2 -k awskey -i
~/.ssh/awskey.pem -r us-west-2 -z us-west-2a --vpc-id=vpc-a28d24c7 -- subnet-id=subnet-4eb27b39 -s 2 -t c4.xlarge -m m4.large --spark-version=1.5.2 --copy-aws- credentials launch my-spark-cluster $ scp -i ~/.ssh/awskey.pem target/scala-2.10 /my_job-assembly-0.1.jar root@<copy-master-ip- by-yourself>:~/job.jar $ ssh -i ~/.ssh/awskey.pem root@<master-ip> '. /spark/bin/spark-submit --class mypackage.Main --master spark://<master-ip>:7077 --executor- memory 6G job.jar arg0' $ spark-ec2 -r us-west-2 destroy my-spark- cluster
spark-ec2 write the code compile & assembly submit job create
cluster destroy cluster sbt spark-ec2 spark-ec2 scp & ssh make
spark-ec2’s bad parts Need to install sbt and spark-ec2. Need
to design and maintain Makefiles. Slow startup time (~20mins).
• spark-ec2 • amazon emr (Elastic MapReduce) • spark-deployer Solutions
emr write the code compile & assembly submit job create
cluster destroy cluster sbt emr
emr’s commands $ sbt assembly $ aws emr create-cluster --name
my-spark-cluster --release-label emr-4.2.0 --instance-type m3. xlarge --instance-count 2 --applications Name=Spark --ec2-attributes KeyName=awskey --use- default-roles $ aws emr put --cluster-id j-2AXXXXXXGAPLF --key- pair-file ~/.ssh/mykey.pem --src target/scala- 2.10/my_job-assembly-0.1.jar --dest /home/hadoop/job.jar $ aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=Spark,Name=my-emr,ActionOnFailure= CONTINUE,Args=[--executor-memory,13G,--class, mypackage.Main,/home/hadoop/job.jar,arg0] $ aws emr terminate-clusters --cluster-id j-2AXX
emr write the code compile & assembly submit job create
cluster destroy cluster sbt emr make
emr’s bad parts Need to install sbt and emr. Need
to design and maintain Makefiles. Spark’s version is old. Restricted machine type.
Since sbt is a powerful build tool itself, why don’t
we let it handle all the dirty works for us?
• spark-ec2 • amazon emr (Elastic MapReduce) • spark-deployer Solutions
spark-deployer write the code compile & assembly submit job create
cluster destroy cluster sbt
spark-deployer’s commands $ sbt "sparkCreateCluster 2" $ sbt "sparkSubmitJob arg0"
$ sbt "sparkDestroyCluster"
spark-deployer’s good parts Need to install only sbt. No Makefile.
Easy to use. Let you focus on your code. Fast and parallel startup (~4mins). Dynamic scale out. Flexible design.
How to use it?
Prerequisites • java • sbt • export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=...
http://www.scala-sbt.org/0.13/tutorial/Manual-Installation.html#Unix sbt installation
Demo
• Report issues. • Join our gitter channel. • Send
pull requests. https://github.com/KKBOX/spark-deployer Give it a try, and share! KKBOX / spark-deployer
Thank you Pishen Tsai @ KKBOX KKBOX / spark-deployer
More Decks by Pishen Tsai
See All by Pishen Tsai
Introduction to Minitime
pishen
1
240
都什麼時代了,你還在寫 while loop 嗎?
pishen
2
790
Pishen's Emacs Journey
pishen
0
170
Scala + Google Dataflow = Serverless Spark
pishen
6
890
Shapeless Introduction
pishen
2
970
ScalaKitchen
pishen
1
500
sbt-emr-spark
pishen
1
180
My Personal Report of Scala Kansai 2016
pishen
0
450
SBT Basic Concepts
pishen
1
690
Other Decks in Programming
See All in Programming
Web3Dライブラリを作ってVue Custom Rendererの仕組みを理解しよう / lets learn vue custom renderer by making 3d library
drumath2237
1
140
作れるエンジニアから、価値を届けるエンジニアへ
ryunakayama
1
310
Enforcing Your Architecture with Unit Tests in the AI Era (Swift Connection X)
steliosf
PRO
1
180
AI駆動開発4年目の現在地(『あなたはFDEになるべきか?』イベント資料)
kumaiu
0
650
Secure Your Coding Agent Like It’s Malware (v1.5) 🇬🇧 @Code.talks 2026
richargh
0
170
[DroidKnights 2026] Workflow AX 적용기 — 자동 정책 정리부터 E2E까지
dayeon
0
100
Rails初心者がパスキーを実装してみて大変だったこと
ktombow
0
210
[DroidKnights 2026] Workflow AX: From Auto-Documented Policies to E2E Testing
raeminkang
0
580
AWS CDKを理解するために公式の定義を深掘って解説してみる / Understanding AWS CDK by Digging into Its Official Definition
akihisaikeda
3
310
エージェントを使い続けるためのメタワーク / Meta-work for keeping using the agent
rkaga
6
2.3k
オーケストレーション記述言語 Katari
yukikurage
0
270
LLM 101
dalinaum
0
500
Featured
See All Featured
We Are The Robots
honzajavorek
0
450
The Cost Of JavaScript in 2023
addyosmani
56
10k
VelocityConf: Rendering Performance Case Studies
addyosmani
331
25k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
techseoconnect
PRO
0
490
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
287
14k
How to Align SEO within the Product Triangle To Get Buy-In & Support - #RIMC
aleyda
2
1.9k
Agile Leadership in an Agile Organization
kimpetersen
PRO
0
310
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
helenjbeal
2
400
Stop Working from a Prison Cell
hatefulcrawdad
274
21k
Unsuck your backbone
ammeep
672
58k
RailsConf 2023
tenderlove
30
1.6k
Believing is Seeing
oripsolob
1
280