30th 2017 val words = sc.textFile("poem.txt") .flatMap(line ⇒ line.split("\\s+")) .map(word ⇒ (word.length, word)) // Number of words by length val wordLen: Map[Int, Long] = words.countByKey()
30th 2017 • Library of Machine Learning • Algorithms of different types: • Classification • Regression • Clustering • Collaborative filtering • … • Implemented using Spark specific features: • Partitioning of the data • Iterative processing performed in memory • Optimized for large volumes of data
30th 2017 • Machine Learning library built on top of RDDs • Specific data structures • Vector: a list of features, can be dense or sparse • LabeledPoint: a vector + a label • In supervised learning: • Training data: RDD[LabeledPoint] • Test data: RDD[Vector] • No longer recommended → Spark ML
30th 2017 • Machine Learning library built on top of DataFrames ➔ New in Spark 1.2 • API with high-level components: ➔ Estimators: they generate a Model from training data ➔ Transformers: they transform the data ➔ A Model is itself a Transformer ➔ A Machine Learning algorithm is an Estimator
30th 2017 • Steps: ➔ On training data, for each stage: if the stage is an Estimator, call its fit() method and the transform() method of the resulting Model; if the stage is a Transformer, call its transform() method ➔ To make predictions on test or new data, call the transform() method of each Transformer (including Models from the training step) • Tedious → use Pipelines
30th 2017 • Spark ML provides an API to build a Pipeline composed of Estimators and Transformers • Steps: ➔ Assemble the Pipeline ➔ Call Pipeline.fit() on training data to get a PipelineModel: the fit() method is called on each Estimator and the transform() method is called on each Transformer (including Models) ➔ Call PipelineModel.transform() on data to make predictions: the transform() method is called on each component of the Pipeline
30th 2017 • Conventions for column names: ➔ The vector of features: “features” ➔ The label in training data: “label” ➔ The predicted label: “prediction” • These column names are used unless they are explicitly specified
30th 2017 Spark ML provides support for Grid Search optimization through k-fold Cross Validation • With k=10, the model is trained and evaluated on a first split of 9/10 of the data for training and 1/10 for validation. • The model is then trained on another split, and so on 10 times. The final accuracy is the average of the accuracies measured for each fold
30th 2017 • Parameters are defined through a ParamGridBuilder • Performance is measured using an Evaluator: ➔ RegressionEvaluator: for numerical predictions ➔ BinaryClassificationEvaluator: for binary predictions ➔ MultiClassClassificationEvaluator: for multi-category predictions • Instantiate a CrossValidator: ➔ Define the number of folds, the Pipeline, the Estimator, the param grid and the Evaluator ➔ Then call fit() to search for the most accurate model
aseigneurin
smt7174
taichinakamura
cyberagentdevelopers
rkaga
shuzon
oracle4engineer
non97
ar_tama
calm1205
hide212131
chrisshort
bkeepers
garrettdimon
productmarketing
holman
panda_program
destraynor
sonatard
michaelherold
palkan
tammyeverts
chrislema
