Lightning-fast Machine Learning with Spark

@nivdul #DV14 #MLwithSpark Lightning fast Machine Learning with Spark Ludwine
Probst

@nivdul #Devoxx #MLwithSpark me Data engineer at Leader of Duchess
France

@nivdul #Devoxx #MLwithSpark Machine Learning

@nivdul #DV14 #MLwithSpark MapReduce Lay of the land

@nivdul #Devoxx #MLwithSpark MapReduce

@nivdul #Devoxx #MLwithSpark HDFS with iterative algorithms

@nivdul #Devoxx #MLwithSpark

@nivdul #Devoxx #MLwithSpark is a fast and general engine for
large-scale data processing

@nivdul #DV14 #MLwithSpark •big data analytics in memory/disk •complements Hadoop
•fast and more flexible •Resilient Distributed Datasets (RDD) •shared variables

@nivdul #Devoxx #MLwithSpark Shared variables broadcast variables accumulators val broadcastVar
= sc.broadcast(Array(1, 2, 3)) val acc = sc.accumulator(0, "MyAccumulator") sc.parallelize(Array(1, 2, 3)).foreach(x => acc += x)

@nivdul #DV14 #MLwithSpark RDD (Resilient Distributed Datasets) •process in parallel
•controllable persistence (memory, disk…) •higher-level operations (transformation & actions) •rebuilt automatically using lineage

@nivdul #Devoxx #MLwithSpark Data Storage InputFormat cassandra cassandra

@nivdul #Devoxx #MLwithSpark Spark data flow

@nivdul #Devoxx #MLwithSpark Languages interactive shell (scala & python) Lambda
(Java 8)

@nivdul #Devoxx #MLwithSpark val conf = new SparkConf() .setAppName("Spark word
count") .setMaster("local") ! val sc = new SparkContext(conf) WordCount example (scala)

@nivdul #DV14 #MLwithSpark // load the data val data =
sc.textFile("filepath/wordcount.txt") // map then reduce step val wordCounts = data.flatMap(line => line.split("\\s+")) .map(word => (word, 1)) .reduceByKey(_ + _) // persist the data wordCounts.cache()

@nivdul #DV14 #MLwithSpark // keep words which appear more than
3 times val filteredWordCount = wordCounts.filter { case (key, value) => value > 2 } ! filteredWordCount.count()

@nivdul #Devoxx #MLwithSpark Spark ecosystem

@nivdul #Devoxx #MLwithSpark streaming makes it easy to build scalable
fault-tolerant streaming applications

@nivdul #Devoxx #MLwithSpark SQL unifies access to structured data

@nivdul #Devoxx #MLwithSpark is Apache Spark's API for graphs and
graph-parallel computation

@nivdul #Devoxx #MLwithSpark MLlib is Apache Spark's scalable machine learning
library

@nivdul #Devoxx #MLwithSpark Machine learning with Spark / MLlib

@nivdul #Devoxx #MLwithSpark Machine learning libraries scikit

@nivdul #Devoxx #MLwithSpark Example make a movies recommender system

@nivdul #Devoxx #MLwithSpark Collaborative filtering with Alternating Least Square (ALS)

@nivdul #DV14 #MLwithSpark 1 3 5 1 28 4 2
18 3 2 5 5 userID movieID rating

@nivdul #DV14 #MLwithSpark // Load and parse the data val
data = sc.textFile("movies.txt") ! // create a RDD[Rating] val ratings = data.map(_.split("\\s+") match { case Array(user, movie, rate) => Rating(user.toInt, movie.toInt, rate.toDouble) })

@nivdul #DV14 #MLwithSpark // split the data into training set
and test set val splits = ratings.randomSplit(Array(0.8, 0.2)) ! // persist the training set val training = splits(0).cache() val test = splits(1)

@nivdul #DV14 #MLwithSpark // Build the recommendation model using ALS
! val model = ALS.train(training, rank = 10, iterations = 20, 1)

@nivdul #DV14 #MLwithSpark // Evaluate the model val userMovies =
test.map { case Rating(user, movie, rate) => (user, movie) } val predictions = model.predict(userMovies).map { case Rating(user, movie, rate) => ((user, movie), rate) } ! val ratesAndPreds = test.map { case Rating(user, movie, rate) => ((user, movie), rate) }.join(predictions) //measuring the Mean Squared Error of rating prediction val MSE = ratesAndPreds.map { case ((user, movie), (r1, r2)) => val err = (r1 - r2) err * err }.mean()

@nivdul #DV14 #MLwithSpark // recommending movies ! val recommendations =
model.recommendProducts(2, 10) .sortBy(- _.rating) ! var i = 1 recommendations.foreach { r => println(r.product + " with rating " + r.rating) i += 1 }

@nivdul #Devoxx #MLwithSpark Performance Spark core Hadoop MapReduce http://databricks.com/blog/2014/10/10/spark-breaks-previous-large-scale-sort-record.html How
fast a system can sort 100 TB of data on disk ?

@nivdul #Devoxx #MLwithSpark Performance Spark / MLlib Collaborative filtering with
MLlib vs Mahout https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html

@nivdul #Devoxx #MLwithSpark Why should I care ? fast and
easy Machine Learning with MLlib fast & flexible in-memory /on-disk SQL Streaming MLlib

Lightning-fast Machine Learning with Spark

Lightning-fast Machine Learning with Spark

More Decks by Probst Ludwine

Other Decks in Programming

Featured

Transcript