ML and the IoT: Living on the Edge

Machine Learning and the IoT are a match made in heaven. After all, IoT devices collect mountains of sensor data, what better way to uncover insights and actions than through sophisticated, modern computing methods like ML and AI?

The problem is, leveraging ML with IoT has historically meant backhauling all your sensor data to the Cloud. When the cloud is involved, security is a concern, and in the realm of IoT, security is often a dirty word.

But modern embedded systems, microcontrollers and single-board computers are getting more powerful, and more sophisticated, and its becoming increasingly possible to bring Machine Learning closer to sensors and IoT devices. "Edge ML" enables quicker insights, tighter security, and even true predictive action, and it's going to become the norm in the IoT in the near future.

In this session, we'll explore the state of the art in Edge ML and IoT, and talk about practical ways that developers can get started with both, today.

Transcript

1. MACHINE LEARNING & THE IOT: LIVING ON THE EDGE BRANDON SATROM | BRANDON@PARTICLE.IO
3. Zeke
4. Zeke
5. To “Zeke”
6. To “Zeke”
7. Zeked!
8. Zeked!
9. Zeked!
10. Zeked!
11. Zeked!
12. Zeked!
13. Zeked!
14. Zeked!
15. Zeked!
30. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING 1 2 DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
31. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING 1 2 DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
32. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING 1 2 DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
33. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
34. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
35. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
36. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
37. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
38. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
39. AI, MACHINE LEARNING & DEEP LEARNING AI Machine learning Representation learning Deep learning Example: Knowledge bases Example: Logistic regression Example: Shallow autoencoders Example: MLPs
40. ALGORITHMS VS. PREDICITONS
41. ALGORITHMS VS. PREDICITONS “Learning” Happens Here
42. DEEP LEARNING: FINDING CORRELATION AMONG COMPLEX DATA
43. DEEP LEARNING AS FEATURE EXTRACTION
44. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask
45. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask Some combination of these three lights tells us when its safe to walk or stop
46. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask Some combination of these three lights tells us when its safe to walk or stop
47. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask Some combination of these three lights tells us when its safe to walk or stop
48. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask Some combination of these three lights tells us when its safe to walk or stop
49. THE STREETLIGHT PROBLEM Examples and images from “Grokking Deep Learning” by Andrew Trask Some combination of these three lights tells us when its safe to walk or stop After a few rounds, a human can infer that the middle light is the key
50. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask
51. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask
52. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask
53. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask
54. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask import numpy as np weights = np.array([0.5, 0.48, -0.7]) alpha = 0.1 streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1], [0, 1, 1],[1, 0, 1]]) walk_vs_stop = np.array([0, 1, 0, 1, 1, 0]) input = streetlights[0] goal_prediction = walk_vs_stop[0] for iteration in range(40): error_for_all_lights = 0 for row_index in range(len(walk_vs_stop)): input = streetlights[row_index] goal_prediction = walk_vs_stop[row_index] prediction = input.dot(weights) error = (goal_prediction - prediction) "** 2 error_for_all_lights += error delta = prediction - goal_prediction weights = weights - (alpha * (input * delta)) print("Prediction:" + str(prediction)) print("Error:" + str(error_for_all_lights) + "\n")
55. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask import numpy as np weights = np.array([0.5, 0.48, -0.7]) alpha = 0.1 streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1], [0, 1, 1],[1, 0, 1]]) walk_vs_stop = np.array([0, 1, 0, 1, 1, 0]) input = streetlights[0] goal_prediction = walk_vs_stop[0] for iteration in range(40): error_for_all_lights = 0 for row_index in range(len(walk_vs_stop)): input = streetlights[row_index] goal_prediction = walk_vs_stop[row_index] prediction = input.dot(weights) error = (goal_prediction - prediction) "** 2 error_for_all_lights += error delta = prediction - goal_prediction weights = weights - (alpha * (input * delta)) print("Prediction:" + str(prediction)) print("Error:" + str(error_for_all_lights) + "\n")
56. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask import numpy as np weights = np.array([0.5, 0.48, -0.7]) alpha = 0.1 streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1], [0, 1, 1],[1, 0, 1]]) walk_vs_stop = np.array([0, 1, 0, 1, 1, 0]) input = streetlights[0] goal_prediction = walk_vs_stop[0] for iteration in range(40): error_for_all_lights = 0 for row_index in range(len(walk_vs_stop)): input = streetlights[row_index] goal_prediction = walk_vs_stop[row_index] prediction = input.dot(weights) error = (goal_prediction - prediction) "** 2 error_for_all_lights += error delta = prediction - goal_prediction weights = weights - (alpha * (input * delta)) print("Prediction:" + str(prediction)) print("Error:" + str(error_for_all_lights) + "\n") Prediction: -0.19999999999999996 Prediction: -0.19999999999999996 Prediction: -0.5599999999999999 Prediction: 0.6160000000000001 Prediction: 0.17279999999999995 Prediction: 0.17552 Error: 2.6561231104 … Prediction: -0.0022410273814405524 Prediction: 0.9978745386023716 Prediction: -0.016721264429884947 Prediction: 1.0151127459893812 Prediction: 0.9969492081270097 Prediction: -0.0026256193329783125 Error:0.00053373677328488
57. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION Examples and images from “Grokking Deep Learning” by Andrew Trask import numpy as np weights = np.array([0.5, 0.48, -0.7]) alpha = 0.1 streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1], [0, 1, 1],[1, 0, 1]]) walk_vs_stop = np.array([0, 1, 0, 1, 1, 0]) input = streetlights[0] goal_prediction = walk_vs_stop[0] for iteration in range(40): error_for_all_lights = 0 for row_index in range(len(walk_vs_stop)): input = streetlights[row_index] goal_prediction = walk_vs_stop[row_index] prediction = input.dot(weights) error = (goal_prediction - prediction) "** 2 error_for_all_lights += error delta = prediction - goal_prediction weights = weights - (alpha * (input * delta)) print("Prediction:" + str(prediction)) print("Error:" + str(error_for_all_lights) + "\n") Prediction: -0.19999999999999996 Prediction: -0.19999999999999996 Prediction: -0.5599999999999999 Prediction: 0.6160000000000001 Prediction: 0.17279999999999995 Prediction: 0.17552 Error: 2.6561231104 … Prediction: -0.0022410273814405524 Prediction: 0.9978745386023716 Prediction: -0.016721264429884947 Prediction: 1.0151127459893812 Prediction: 0.9969492081270097 Prediction: -0.0026256193329783125 Error:0.00053373677328488
58. Machine Learning is Really Human Teaching
59. MACHINE LEARNING IS HUMAN TEACHING 1 2 3 CREATE INPUTS AND OUTPUTS DEFINE A MODEL ARCHITECTURE TRAIN, EVALUATE, TUNE CHAPTER 5. MACHINE LEARNING BASICS 100 101 102 103 104 105 Number of training examples 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Error (MSE) Bayes error Train (quadratic) Test (quadratic) Test (optimal capacity) Train (optimal capacity) 10 15 20 ty (polynomial degree)
60. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
61. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
62. MACHINE LEARNING ISN’T “DONE” IN ANY ONE PLACE
63. ML = DEVELOPMENT, TRAINING, INFERENCE 1 2 3 DEVELOPMENT TRAINING INFERENCE
64. MODEL CREATION TOOLS
65. TRAINING, EXPERIMENTS, & TUNING
66. DEMO TRAINING AN EMOTION MODEL
67. THE TRAIN, PREDICT, TUNE LOOP
68. THE TRAIN, PREDICT, TUNE LOOP Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model
69. INFERENCE: PREDICTION IN REAL TIME
70. INFERENCE: PREDICTION IN REAL TIME During training, the network “learns” by tuning weights to reduce error
71. INFERENCE: PREDICTION IN REAL TIME During training, the network “learns” by tuning weights to reduce error These weights are then applied against data the network has never seen before to make a prediction
72. INFERENCE: PREDICTION IN REAL TIME During training, the network “learns” by tuning weights to reduce error These weights are then applied against data the network has never seen before to make a prediction
73. DEMO RUNNING INFERENCE ON A MODEL
74. MODEL DEPLOYMENT + + +
75. PRE-TRAINED MODELS AS APIS
76. DEMO USING THE AZURE FACE API
77. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model
78. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Training & Validation Data
79. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Cloud Training & Deployment Training & Validation Data
80. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Cloud Training & Deployment Training & Validation Data Real time Inference Data ?
81. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
82. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
83. TRADITIONAL: DEVICES AND CLOUD
84. TRADITIONAL: DEVICES AND CLOUD The Cloud The Internet of things
85. TRADITIONAL: DEVICES AND CLOUD The Cloud The Internet of things “Lots of power”
86. TRADITIONAL: DEVICES AND CLOUD The Cloud The Internet of things “Lots of power” “Lots of Devices”
87. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things “Lots of power” “Lots of Devices”
88. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things The “Edge” “Lots of power” “Lots of Devices”
89. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things The “Edge” “Lots of power” “Lots of Devices”
90. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things The “Edge” “Lots of power” “Lots of Devices”
91. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things The “Edge” “Lots of power” “Lots of Devices”
92. THE CLOUD, THE “EDGE,” AND THE IOT The Cloud The Internet of things The “Edge” “Lots of power” “Lots of Devices” “A Bit of both”
93. THE “EDGE” IS THE INTERNET OF THINGS! The Cloud The Internet of things The “Edge”
94. THE “EDGE” IS THE INTERNET OF THINGS! The Cloud The Internet of things The “Edge”
95. THE IOT IS THE SOURCE OF DATA FOR ML Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Cloud Training & Deployment Training & Validation Data Real time Inference Data ?
96. THE IOT IS THE SOURCE OF DATA FOR ML Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Cloud Training & Deployment Training & Validation Data Real time Inference Data ?
97. THE IOT IS THE SOURCE OF DATA FOR ML Define Task Collect Data Prepare Data Test & Evaluate Train Model Deploy Model Maintain, Update & Refine Tune & Refine Explore Model Local Development Cloud Training & Deployment Training & Validation Data Real time Inference Data ?
98. WHY DO ML INFERENCING IN THE IOT? 1 2 3 VOLUME SPEED PRIVACY
99. MORE DEVICES == MORE DATA
103. The Cloud may be infinite; your pipe to it is not
104. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES
105. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES
106. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD
107. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD
108. INFERENCE FOR PREVENTATIVE MAINTENANCE
109. INFERENCE FOR PREVENTATIVE MAINTENANCE
110. INFERENCE FOR PREVENTATIVE MAINTENANCE
111. INFERENCE FOR PREVENTATIVE MAINTENANCE
112. INFERENCE FOR PREVENTATIVE MAINTENANCE
113. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE
114. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE
115. INFERENCE FOR PREVENTATIVE MAINTENANCE
116. INFERENCE FOR PREVENTATIVE MAINTENANCE
117. INFERENCE FOR PREVENTATIVE MAINTENANCE
118. IoT Inferencing == Built-In Privacy
119. VISION IN THE CLOUD
120. VISION IN THE CLOUD
121. VISION AT THE EDGE
122. VISION AT THE EDGE
123. VISION AT THE EDGE [ { "faceId": "b0f2804a-649f-4da3-b6cf-ff459c867ae8", "faceRectangle": { "width": 63, "left": 168, "height": 63, "top": 110 }, "faceAttributes": { "smile": 0.0, "exposure": { "exposureLevel": "overExposure", "value": 0.75 }, "emotion": { "sadness": 0.0, "surprise": 0.0, "neutral": 0.0, "contempt": 0.001, "anger": 0.666, "disgust": 0.333, "fear": 0.0, "happiness": 0.0 } "facialHair": { "beard": 0.4, "moustache": 0.4, "sideburns": 0.4 }, "gender": "male", "age": 33.0, "headPose": { "yaw": -24.3, "pitch": -6.4, "roll": -8.1 } } } ]
124. EDGE ML INFERENCING DEVICES 1 2 GOOGLE CORAL NVIDIA JETSON NANO
125. GOOGLE CORAL • Edge TPU Module (SOM) • NXP i.MX 8M SOC (Quad-core Cortex-A53, plus Cortex-M4F) • Google Edge TPU ML accelerator coprocessor • Wi-Fi 2x2 MIMO (802.11b/g/n/ac 2.4/5GHz) • Bluetooth 4.1 • 8GB eMMC • 1GB LPDDR4 • USB connections • USB Type-C power port (5V DC) • USB 3.0 Type-C OTG port • USB 3.0 Type-A host port • USB 2.0 Micro-B serial console port • Audio & Video connections • Digital PDM microphone (x2) • HDMI 2.0a (full size) • 39-pin FFC connector for MIPI DSI display (4-lane) • 24-pin FFC connector for MIPI CSI-2 camera (4-lane) • MicroSD card slot • Gigabit Ethernet port • 40-pin GPIO expansion header • Supports Mendel Linux (derivative of Debian)
126. NVIDIA JETSON NANO • GPU and CPU • 128 Core Maxwell (472 GFLOPs) • Quad-core Cortex-A57 @ 1.43 GHz • 4 GB 64 bit LPDDR4 25.6 GB/s • 16GB eMMC • Video • HDMI 2.0 or DP1.2 • 4K Encode and Decode • 12-pin FFC connector for MIPI CSI-2 camera (1.1-lane) • Connections • 1x SDIO, 2x SPI, 5x SysIO, 13x GPIOs, 6x I2C • USB connections • 1 USB 3.0 • MicroSD card slot • Gigabit Ethernet port • Supports Ubuntu Linux
127. DEMO FACE DETECTION WITH GOOGLE CORAL
128. Where does this leave the Internet of Things?
129. THE INCREASING ROLE OF THE IOT IN ML 1 2 COLLECTION & COORDINATION ON DEVICES ML INFERENCING ON MICROCONTROLLERS!
131. ML data doesn’t come from the cloud, or the edge… It comes from the IoT
132. IOT DEVICES SENSE AND SHARE
133. IOT DEVICES SENSE AND SHARE
134. IOT DEVICES TAKE ACTION
135. IOT DEVICES TAKE ACTION
136. IOT DEVICES TAKE ACTION
137. IOT DEVICES TAKE ACTION
138. Edge ML + Mesh Networking = Crazy Delicious
139. TRADITIONAL IOT: HUB AND SPOKE Wi-Fi Router
140. TRADITIONAL IOT: HUB AND SPOKE Wi-Fi Router
141. TRADITIONAL IOT: HUB AND SPOKE Wi-Fi Router
142. TRADITIONAL IOT: HUB AND SPOKE
143. ENTER MESH NETWORKING Gateway
144. ENTER MESH NETWORKING Gateway
145. ENTER MESH NETWORKING Gateway
146. ENTER MESH NETWORKING Gateway
147. ENTER MESH NETWORKING Gateway
148. ENTER MESH NETWORKING
149. OPENTHREAD VS. ZIGBEE, ZWAVE AND LORA Operating range 100 ft 35 ft Max # of devices 232 65,000 Data rate 9.6-100 kbps 40-250 kbps Cost $ $$ IP-Based Networking No No Open Standard? No Yes 1-4 mi 100 ft N/A 300 27 kbps 250 kbps $$$ $ Yes Yes No Yes
150. WHAT IS THREAD? ✴IPv6-based mesh ✴Wireless Personal Area Network ✴No single point of failure ✴Tailored to IoT Scenarios ✴Can be used in concert with Wi-Fi, Cellular and Bluetooth is a low-power networking protocol
151. PARTICLE MESH DEVICES Nordic nRF52840 SoC ✴ ARM Cortex-M4F 32-bit ✴ 1MB flash, 256KB RAM ✴ IEEE 802.15.4-2006 ✴ Bluetooth 5: 2 Mbps, 1 Mbps, 500 Kbps, 125 Kbps ✴ ARM TrustZone Cryptographic security module ✴ NFC-A tag Argon » Wi-Fi + BLE +Mesh » Wi-Fi endpoint or mesh gateway Xenon » BLE + Mesh » Mesh endpoint Boron » LTE-M1 + BLE + Mesh » Cellular endpoint or mesh gateway
152. BRINGING AN EDGE DEVICE INTO A MESH NETWORK The Internet of things
153. BRINGING AN EDGE DEVICE INTO A MESH NETWORK The Internet of things
154. BRINGING AN EDGE DEVICE INTO A MESH NETWORK The Internet of things We can bring an Edge ML device into an IoT mesh network
155. DEMO EMOTION MESH
156. THE FUTURE OF ML AND THE IOT 1 2 ML INFERENCING ON MICROCONTROLLERS! FEDERATED AND DISTRIBUTED LEARNING
157. TENSORFLOW LITE MICRO
158. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES
159. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES
160. TF LITE + MICROTENSOR == A ROBUST, OPEN APPROACH TO MCU ML INFERENCING
161. DEMO ML ON MCUS
162. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
163. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES IoT Devices spend most of their time idle…
164. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES IoT Devices spend most of their time idle… Federated learning may enable models to be distributed between the cloud, edge and devices for the most efficient processing
165. OTA MODEL UPDATES!
166. OTA MODEL UPDATES!
167. OTA MODEL UPDATES!
168. THREE BIG IDEAS 1 2 3 MACHINE LEARNING IS HUMAN TEACHING TRAINING IS MEANT FOR THE CLOUD THE REAL WORK OF ML IS IN THE IOT
169. Explore ML inferencing with SBCs
170. Explore Mesh networking for local network intelligence
171. Explore TensorFlow Lite and µTensor for ML on MCUs
172. THANK YOU! BRANDON SATROM | BRANDON@PARTICLE.IO GITHUB.COM/BSATROM/EMOTION-MESH