Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Machine Learning with Clojure and Apache Spark
Search
Eric Weinstein
October 25, 2016
Technology
1
410
Machine Learning with Clojure and Apache Spark
Slides for my EuroClojure 2016 talk on machine learning.
Eric Weinstein
October 25, 2016
Tweet
Share
More Decks by Eric Weinstein
See All by Eric Weinstein
Interview Them Where They Are
ericqweinstein
0
120
Value Your Types!
ericqweinstein
0
90
Being Good: An Introduction to Robo- and Machine Ethics
ericqweinstein
1
1.9k
What If...?: Ruby 3
ericqweinstein
1
210
Infinite State Machine
ericqweinstein
1
120
Do Androids Dream of Electronic Dance Music?
ericqweinstein
1
100
Machine Learning with Elixir and Phoenix
ericqweinstein
1
960
Domo Arigato, Mr. Roboto: Machine Learning with Ruby
ericqweinstein
1
1.5k
A Nil Device, A Lonely Operator, and a Voyage to the Void Star
ericqweinstein
1
990
Other Decks in Technology
See All in Technology
新アイテムをどう使っていくか?みんなであーだこーだ言ってみよう / 20250911-rpi-jam-tokyo
akkiesoft
0
310
これでもう迷わない!Jetpack Composeの書き方実践ガイド
zozotech
PRO
0
1k
いま注目のAIエージェントを作ってみよう
supermarimobros
0
340
【NoMapsTECH 2025】AI Edge Computing Workshop
akit37
0
220
AI時代を生き抜くエンジニアキャリアの築き方 (AI-Native 時代、エンジニアという道は 「最大の挑戦の場」となる) / Building an Engineering Career to Thrive in the Age of AI (In the AI-Native Era, the Path of Engineering Becomes the Ultimate Arena of Challenge)
jeongjaesoon
0
210
JTCにおける内製×スクラム開発への挑戦〜内製化率95%達成の舞台裏/JTC's challenge of in-house development with Scrum
aeonpeople
0
250
下手な強制、ダメ!絶対! 「ガードレール」を「檻」にさせない"ガバナンス"の取り方とは?
tsukaman
2
450
はじめてのOSS開発からみえたGo言語の強み
shibukazu
3
890
共有と分離 - Compose Multiplatform "本番導入" の設計指針
error96num
2
1k
新規プロダクトでプロトタイプから正式リリースまでNext.jsで開発したリアル
kawanoriku0
1
160
Agile PBL at New Grads Trainings
kawaguti
PRO
1
440
Unlocking the Power of AI Agents with LINE Bot MCP Server
linedevth
0
110
Featured
See All Featured
The Cult of Friendly URLs
andyhume
79
6.6k
Six Lessons from altMBA
skipperchong
28
4k
Bash Introduction
62gerente
615
210k
Statistics for Hackers
jakevdp
799
220k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
Side Projects
sachag
455
43k
Automating Front-end Workflow
addyosmani
1370
200k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
810
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
13k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Transcript
Machine Learning with Clojure and Apache Spark ;; Eric Weinstein
;; EuroClojure 2016 ;; Bratislava, Slovakia ;; 25 October 2016
for Joshua
Part 0: Hello!
About Me (def eric-weinstein {:employer "Hulu" :github "ericqweinstein" :twitter "ericqweinstein"
:website "ericweinste.in"}) 30% off with EURORUBY30!
Agenda • Machine learning • Apache Spark • Flambo vs.
Sparkling • DL4J, deep learning, and convolutional neural networks
Part 1: ⚡✨
What’s machine learning?
In a word:
Generalization
What’s Supervised Learning? Classification or regression, generalizing from labeled data
to unlabeled data
What’s Apache Spark? Apache Spark is an open-source cluster computing
framework; its parallelism makes it ideal for processing large data sets, and in ML, the more data, the better!
Some Spark Terminology • RDD: Resilient Distributed Dataset • Dataset:
RDD + Spark SQL execution engine • DataFrame: Dataset organized into named columns
Our Data • Police stop data for the city of
Los Angeles, California in 2015 • 4 features, ~600,000 instances • http://bit.ly/2f9jVwn
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Features && Labels • Sex (Male | Female) • Race
(American Indian | Asian | Black | Hispanic | White | Other) • Stop type (Pedestrian | Vehicle) • Post-stop activity (Yes | No)
Decision Trees X[0] <= 0.5 gini = 0.4033 samples =
139572 value = [100477, 39095] X[1] <= 5.5 gini = 0.4318 samples = 102419 value = [70118, 32301] True X[1] <= 5.5 gini = 0.2989 samples = 37153 value = [30359, 6794] False X[1] <= 4.5 gini = 0.4399 samples = 96665 value = [65083, 31582] gini = 0.2187 samples = 5754 value = [5035, 719] X[1] <= 3.5 gini = 0.4483 samples = 78400 value = [51805, 26595] gini = 0.397 samples = 18265 value = [13278, 4987] X[1] <= 2.5 gini = 0.4324 samples = 51662 value = [35328, 16334] gini = 0.473 samples = 26738 value = [16477, 10261] X[1] <= 0.5 gini = 0.4406 samples = 48927 value = [32894, 16033] gini = 0.1959 samples = 2735 value = [2434, 301] gini = 0.4658 samples = 65 value = [41, 24] gini = 0.4406 samples = 48862 value = [32853, 16009] X[1] <= 3.5 gini = 0.3067 samples = 34817 value = [28234, 6583] gini = 0.1643 samples = 2336 value = [2125, 211] X[1] <= 2.5 gini = 0.2796 samples = 15786 value = [13133, 2653] X[1] <= 4.5 gini = 0.3277 samples = 19031 value = [15101, 3930] X[1] <= 0.5 gini = 0.2921 samples = 13985 value = [11501, 2484] gini = 0.1701 samples = 1801 value = [1632, 169] gini = 0.426 samples = 26 value = [18, 8] gini = 0.2918 samples = 13959 value = [11483, 2476] gini = 0.3747 samples = 9522 value = [7144, 2378] gini = 0.2732 samples = 9509 value = [7957, 1552]
Part 2: A Tale of Two DSLs vs. ✨✨ Image
credit: Adventure Time
Flambo Example (defn make-spark-context "Creates the Apache Spark context using
the Flambo DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (f/spark-context)))
Sparkling Example (defn make-spark-context "Creates the Apache Spark context using
the Sparkling DSL." [] (-> (conf/spark-conf) (conf/master "local") (conf/app-name "euroclojure") (spark/spark-context)))
Straight Spark (def model (DecisionTree/trainClassifier training 2 categorical-features- info "gini"
5 32)) ; max depth: 5, max leaves: 32 (defn predict [p] ; LabeledPoint (let [prediction (.predict model (.features p))] [(.label p) prediction]))
Accuracy: 0.77352
Part 3: Deep Learning
What’s Deep Learning? • Neural networks (computational architecture modeled after
the human brain) • Neural networks with many layers (> 1 hidden layer, but in practice, can be hundreds) • The vanishing/exploding gradient problem
Vanishing && Gradients
Image credit for all ConvNet images: https://deeplearning4j.org/convolutionalnets
Max Pooling/Downsampling
Alternating Layers
Our Data Image credit: http://digitalmedia.fws.gov/cdm/
What’s DL4J? • DL4J == Deep Learning 4 Java, a
library (for Java, unsurprisingly) • Examples on GitHub: https://github.com/ deeplearning4j/deeplearning4j • ConvNet worked example: http://bit.ly/2eBM8ss
DL4J Example (def nn-conf (-> (NeuralNetConfiguration$Builder.) ;; Some values omitted
for space (.activation "relu") (.learningRate 0.0001) (.weightInit (WeightInit/XAVIER)) (.optimizationAlgo OptimizationAlgorithm/STOCHASTIC_GRADIENT_DESCENT) (.updater Updater/RMSPROP) (.momentum 0.9) (.list) (.layer 0 conv-init) (.layer 1 (max-pool "maxpool1" (int-array [2 2]))) (.layer 2 (conv-5x5 "cnn2" 100 (int-array [5 5]) (int-array [1 1]) 0)) (.layer 3 (max-pool "maxpool2" (int-array [2 2]))) (.layer 4 (fully-connected 500)) (.layer 5 output-layer) (.build)))
How’d We Do? • Accuracy: 0.375 • Precision: 0.3333 •
Recall: 0.375 • F1 Score: 0.3529
Summary • Clojure + Spark = • Flambo and Sparkling
are roughly equally powerful • Deep learning is super doable with Clojure (though Java interop is kind of a pain)
Takeaways (TL;DPA) • Contribute to Flambo and/or Sparkling! • Let’s
build or contribute to a nicer DSL for DL4J • https://github.com/ericqweinstein/euroclojure
None