Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Machine Learning with Clojure and Apache Spark
Search
Eric Weinstein
October 25, 2016
Technology
1
400
Machine Learning with Clojure and Apache Spark
Slides for my EuroClojure 2016 talk on machine learning.
Eric Weinstein
October 25, 2016
Tweet
Share
More Decks by Eric Weinstein
See All by Eric Weinstein
Interview Them Where They Are
ericqweinstein
0
110
Value Your Types!
ericqweinstein
0
86
Being Good: An Introduction to Robo- and Machine Ethics
ericqweinstein
1
1.8k
What If...?: Ruby 3
ericqweinstein
1
200
Infinite State Machine
ericqweinstein
1
120
Do Androids Dream of Electronic Dance Music?
ericqweinstein
1
97
Machine Learning with Elixir and Phoenix
ericqweinstein
1
940
Domo Arigato, Mr. Roboto: Machine Learning with Ruby
ericqweinstein
1
1.5k
A Nil Device, A Lonely Operator, and a Voyage to the Void Star
ericqweinstein
1
970
Other Decks in Technology
See All in Technology
推し書籍📚 / Books and a QA Engineer
ak1210
0
140
An introduction to Claude Code SDK
choplin
2
1k
QuickSight SPICE の効果的な運用戦略~S3 + Athena 構成での実践ノウハウ~/quicksight-spice-s3-athena-best-practices
emiki
0
290
Amazon SNSサブスクリプションの誤解除を防ぐ
y_sakata
3
190
モニタリング統一への道のり - 分散モニタリングツール統合のためのオブザーバビリティプロジェクト
niftycorp
PRO
1
520
Delegating the chores of authenticating users to Keycloak
ahus1
0
190
Introduction to Bill One Development Engineer
sansan33
PRO
0
260
毎晩の 負荷試験自動実行による効果
recruitengineers
PRO
5
180
本当にわかりやすいAIエージェント入門
segavvy
1
300
Microsoft Defender XDRで疲弊しないためのインシデント対応
sophiakunii
1
310
SREの次のキャリアの道しるべ 〜SREがマネジメントレイヤーに挑戦して、 気づいたこととTips〜
coconala_engineer
1
4.3k
AIエージェントが書くのなら直接CloudFormationを書かせればいいじゃないですか何故AWS CDKを使う必要があるのさ
watany
18
7.6k
Featured
See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
GraphQLとの向き合い方2022年版
quramy
49
14k
Statistics for Hackers
jakevdp
799
220k
Side Projects
sachag
455
42k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
How to train your dragon (web standard)
notwaldorf
96
6.1k
YesSQL, Process and Tooling at Scale
rocio
173
14k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
How STYLIGHT went responsive
nonsquared
100
5.6k
KATA
mclloyd
30
14k
Transcript
Machine Learning with Clojure and Apache Spark ;; Eric Weinstein
;; EuroClojure 2016 ;; Bratislava, Slovakia ;; 25 October 2016
for Joshua
Part 0: Hello!
About Me (def eric-weinstein {:employer "Hulu" :github "ericqweinstein" :twitter "ericqweinstein"
:website "ericweinste.in"}) 30% off with EURORUBY30!
Agenda • Machine learning • Apache Spark • Flambo vs.
Sparkling • DL4J, deep learning, and convolutional neural networks
Part 1: ⚡✨
What’s machine learning?
In a word:
Generalization
What’s Supervised Learning? Classification or regression, generalizing from labeled data
to unlabeled data
What’s Apache Spark? Apache Spark is an open-source cluster computing
framework; its parallelism makes it ideal for processing large data sets, and in ML, the more data, the better!
Some Spark Terminology • RDD: Resilient Distributed Dataset • Dataset:
RDD + Spark SQL execution engine • DataFrame: Dataset organized into named columns
Our Data • Police stop data for the city of
Los Angeles, California in 2015 • 4 features, ~600,000 instances • http://bit.ly/2f9jVwn
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Decision Trees X[0] <= 0.5 gini = 0.4033 samples =
139572 value = [100477, 39095] X[1] <= 5.5 gini = 0.4318 samples = 102419 value = [70118, 32301] True X[1] <= 5.5 gini = 0.2989 samples = 37153 value = [30359, 6794] False X[1] <= 4.5 gini = 0.4399 samples = 96665 value = [65083, 31582] gini = 0.2187 samples = 5754 value = [5035, 719] X[1] <= 3.5 gini = 0.4483 samples = 78400 value = [51805, 26595] gini = 0.397 samples = 18265 value = [13278, 4987] X[1] <= 2.5 gini = 0.4324 samples = 51662 value = [35328, 16334] gini = 0.473 samples = 26738 value = [16477, 10261] X[1] <= 0.5 gini = 0.4406 samples = 48927 value = [32894, 16033] gini = 0.1959 samples = 2735 value = [2434, 301] gini = 0.4658 samples = 65 value = [41, 24] gini = 0.4406 samples = 48862 value = [32853, 16009] X[1] <= 3.5 gini = 0.3067 samples = 34817 value = [28234, 6583] gini = 0.1643 samples = 2336 value = [2125, 211] X[1] <= 2.5 gini = 0.2796 samples = 15786 value = [13133, 2653] X[1] <= 4.5 gini = 0.3277 samples = 19031 value = [15101, 3930] X[1] <= 0.5 gini = 0.2921 samples = 13985 value = [11501, 2484] gini = 0.1701 samples = 1801 value = [1632, 169] gini = 0.426 samples = 26 value = [18, 8] gini = 0.2918 samples = 13959 value = [11483, 2476] gini = 0.3747 samples = 9522 value = [7144, 2378] gini = 0.2732 samples = 9509 value = [7957, 1552]
Part 2: A Tale of Two DSLs vs. ✨✨ Image
credit: Adventure Time
Flambo Example (defn make-spark-context "Creates the Apache Spark context using
the Flambo DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (f/spark-context)))
Sparkling Example (defn make-spark-context "Creates the Apache Spark context using
the Sparkling DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (spark/spark-context)))
Straight Spark (def model (DecisionTree/trainClassifier training 2 categorical-features- info "gini"
5 32)) ; max depth: 5, max leaves: 32 (defn predict [p] ; LabeledPoint (let [prediction (.predict model (.features p))] [(.label p) prediction]))
Accuracy: 0.77352
Part 3: Deep Learning
What’s Deep Learning? • Neural networks (computational architecture modeled after
the human brain) • Neural networks with many layers (> 1 hidden layer, but in practice, can be hundreds) • The vanishing/exploding gradient problem
Vanishing && Gradients
Image credit for all ConvNet images: https://deeplearning4j.org/convolutionalnets
Max Pooling/Downsampling
Alternating Layers
Our Data Image credit: http://digitalmedia.fws.gov/cdm/
What’s DL4J? • DL4J == Deep Learning 4 Java, a
library (for Java, unsurprisingly) • Examples on GitHub: https://github.com/ deeplearning4j/deeplearning4j • ConvNet worked example: http://bit.ly/2eBM8ss
DL4J Example (def nn-conf (-> (NeuralNetConfiguration$Builder.) ;; Some values omitted
for space (.activation "relu") (.learningRate 0.0001) (.weightInit (WeightInit/XAVIER)) (.optimizationAlgo OptimizationAlgorithm/STOCHASTIC_GRADIENT_DESCENT) (.updater Updater/RMSPROP) (.momentum 0.9) (.list) (.layer 0 conv-init) (.layer 1 (max-pool "maxpool1" (int-array [2 2]))) (.layer 2 (conv-5x5 "cnn2" 100 (int-array [5 5]) (int-array [1 1]) 0)) (.layer 3 (max-pool "maxpool2" (int-array [2 2]))) (.layer 4 (fully-connected 500)) (.layer 5 output-layer) (.build)))
How’d We Do? • Accuracy: 0.375 • Precision: 0.3333 •
Recall: 0.375 • F1 Score: 0.3529
Summary • Clojure + Spark = • Flambo and Sparkling
are roughly equally powerful • Deep learning is super doable with Clojure (though Java interop is kind of a pain)
Takeaways (TL;DPA) • Contribute to Flambo and/or Sparkling! • Let’s
build or contribute to a nicer DSL for DL4J • https://github.com/ericqweinstein/euroclojure
None