Building Camera Features in a Snap

Android 5.0 introduced the Camera2 APIs which include a full spectrum of camera features and configurations. However, building features with this powerful API can be overwhelming.

In this talk, we’ll:

Learn how camera hardware works and how that’s exposed in software
Digest the bits and bytes of color spaces and image formats
Visualize how image kernels work
Set up a testing environment that allows us to iterate quickly
Introduce the concepts of computer vision

This talk will get you up and running with the camera APIs. You’ll learn how to write efficient code that operates on millions of pixels.

Transcript

1. SNAP SNAP SNAP Matt Precious Building Camera Features in a SNAP
2. Adding a Camera Feature to your App
3. How much work do you want to do?
4. How much work do you want to do? Hours Months
5. Hours Months Camera Intent
6. Camera Intent Hours Months
7. Camera Intent Hours Months • Easy
8. Camera Intent Hours Months • Easy • No control over UI
9. Camera Intent Hours Months • Easy • No control over UI • No control over hardware
10. Hours Months
11. Use a Library Hours Months
12. Use a Library Hours Months
13. Library Hours Months
14. Library Hours Months • A lot of code is written for you
15. Library Hours Months • A lot of code is written for you • Changes and fixes require understanding all the code
16. github.com/Fotoapparat/Fotoapparat github.com/google/cameraview github.com/afollestad/material-camera
17. Hours Months
18. Hours Months DIY
19. Hours Months DIY
20. Hours Months DIY
21. Hours Months DIY • Full control over everything
22. Hours Months DIY • Full control over everything • You need to write everything
23. Cameras.
24. Cameras. How do they work?
39. Color Formats
40. http://bit.ly/2jF7Ths
41. RGB
42. Red Green Blue
43. Red Green Blue
44. Red Green Blue
45. RGBA
46. RGB + Alpha
47. RGB A
48. YUV
51. YUV
52. Y'CbCr
53. Y' Cb Cr
54. Y' Cb Cr Luma (Brightness)
55. Y' Cb Cr Chrominance (Color)
56. Y' Cb Cr
57. Y' Cb Cr
58. Y' Cb Cr
59. Y' Cb Cr
60. Y' Cb Cr
61. Why YUV?
66. RGB
67. RGB 888
68. RGB 888 Red Green Blue 8 bits 8 bits 8 bits
69. RGB 888 Red Green Blue 24 bits per pixel 8 bits 8 bits 8 bits
70. RGB 888 Red Green Blue 24 bits per pixel 8 bits 8 bits 8 bits 1920×1080 ≈ 6MB
71. RGB 888 Red Green Blue 24 bits per pixel 8 bits 8 bits 8 bits 1920×1080 ≈ 6MB 60fps: 360MB/s
72. RGB 888
73. RGB 444
74. RGB 444 Red Green Blue 4 bits 4 bits 4 bits
75. RGB 444 Red Green Blue 12 bits per pixel 4 bits 4 bits 4 bits
76. RGB 444 Red Green Blue 12 bits per pixel 4 bits 4 bits 4 bits 1920×1080 ≈ 3MB
77. RGB 444 Red Green Blue 12 bits per pixel 4 bits 4 bits 4 bits 1920×1080 ≈ 3MB 60fps: 180MB/s
78. RGB 444
79. RGB 565
80. RGB 565 16 bits per pixel
81. RGB 565 Red Green Blue 16 bits per pixel 5 bits 5 bits 5 bits
82. RGB 565 Red Green Blue 16 bits per pixel 5 bits 6 bits 5 bits
84. CIE 1931
85. CIE 1931
86. YUV
87. YUV 4:4:4
88. YUV 4:4:4
89. YUV 4:4:4
90. YUV 4:4:4
91. YUV 4:4:4 Y' Cr Cb 8 bits 8 bits 8 bits
92. YUV 4:4:4 Y' Cr Cb 24 bits per pixel 8 bits 8 bits 8 bits
93. YUV 4:4:4 Y' Cr Cb 24 bits per pixel 8 bits 8 bits 8 bits 1920×1080 ≈ 6MB
94. YUV 4:4:4 4:4
95. YUV 4:4:4 2:2
96. YUV 4:2:2
97. YUV 4:2:2
98. YUV 4:2:2
99. YUV 4:2:2
100. YUV 4:2:2 Y' Cr Cb 8 bits 8 bits 8 bits 2 pixels 2 pixels
101. YUV 4:2:2 Y' Cr Cb 8 bits 4 bits 4 bits
102. YUV 4:2:2 Y' Cr Cb 16 bits per pixel 8 bits 4 bits 4 bits
103. YUV 4:2:2 Y' Cr Cb 16 bits per pixel 8 bits 4 bits 4 bits 1920×1080 ≈ 4MB
104. YUV 4:2:2 2
105. YUV 4:2:2 0
106. YUV 4:2:0
107. YUV 4:2:0
108. YUV 4:2:0
109. YUV 4:2:0
110. YUV 4:2:0
111. YUV 4:2:0 Y' Cr Cb 8 bits 8 bits 8 bits 4 pixels 4 pixels
112. YUV 4:2:0 Y' Cr Cb 8 bits 2 bits 2 bits
113. YUV 4:2:0 Y' Cr Cb 12 bits per pixel 8 bits 2 bits 2 bits
114. YUV 4:2:0 Y' Cr Cb 12 bits per pixel 8 bits 2 bits 2 bits 1920×1068 ≈ 3MB
115. Why YUV?
117. http://bit.ly/2fwAxwF
118. Implementing Camera APIs
119. Camera1
120. Camera1 Camera2
121. Camera1 Camera2 Min SDK 1 Min SDK 21 2009 2014
122. Camera1 Camera2 Min SDK 1 Min SDK 21 2009 2014 100% of devices 80%* of devices
123. Camera1 Camera2 Min SDK 1 Min SDK 21 2009 2014 100% of devices 80%* of devices
124. Camera2
125. http://bit.ly/2ylAfAs
127. Permission?
128. Permission? Y
129. Permission? Y What cameras do you have?
130. Permission? Y What cameras do you have? 1, 2
131. Y What cameras do you have? 1, 2
132. Tell me more about camera 1 Y What cameras do you have? 1, 2
133. Tell me more about camera 1 Y What cameras do you have? 1, 2
134. Tell me more about camera 1 1, 2
135. Tell me more about camera 2 Tell me more about camera 1 1, 2
136. Tell me more about camera 2 Tell me more about camera 1 1, 2
137. Tell me more about camera 2
138. Open camera 2, please Tell me more about camera 2
139. Open camera 2, please OK Tell me more about camera 2
140. Open camera 2, please OK
141. Start capturing, please Open camera 2, please OK
142. Start capturing, please xOKx Open camera 2, please OK
143. Start capturing, please xOKx OK
144. Start capturing, please xOKx OK
145. Start capturing, please xOKx OK
146. xOKx
147. Capture xOKx
148. Capture Capture xOKx
149. Capture Capture xOKx
150. github.com/Fotoapparat/Fotoapparat github.com/google/cameraview github.com/afollestad/material-camera github.com/googlesamples/android-Camera2Basic
151. Strides
152. Strides width: 3 height: 3 1 2 3 4 5 6 7 8 9
153. Strides width: 3 height: 3 1 2 3 4 5 6 7 8 9
154. Strides width: 3 height: 3 row stride 1 2 3 4 5 6 7 8 9
155. Strides width: 3 height: 3 row stride: 3 1 2 3 4 5 6 7 8 9
156. Strides width: 3 height: 3 row stride: 5 1 2 3 4 5 6 7 8 9
157. Strides width: 3 height: 3 row stride: 5 1 2 3 4 5 6 7 8 9
158. Strides width: 3 height: 3 row stride: 5 pixel stride 1 2 3 4 5 6 7 8 9
159. Strides width: 3 height: 3 row stride: 5 pixel stride: 1 1 2 3 4 5 6 7 8 9
160. Strides width: 3 height: 3 row stride: 5 pixel stride: 2 1 2 3 4 5 6 7 8 9
161. Strides width: 3 height: 3 row stride: 5 pixel stride: 2 1 2 3 4 5 6 7 8 9
162. Strides width: 3 height: 3 row stride: 7 pixel stride: 2 1 2 3 4 5 6 7 8 9
163. Strides width: 3 height: 3 row stride: 7 pixel stride: 2 1 2 3 4 5 6 7 8 9
164. Kernels
165. Edge Detection
166. Edge Detection 0 1 0 1 –4 1 0 1 0
167. Convolution
168. Convolution
169. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Convolution
170. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Convolution
171. Convolution 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255
172. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 Convolution
173. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 Convolution
174. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 1 0 1 –4 1 0 1 0 Convolution
175. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 1 0 1 –4 1 0 1 0 Convolution
176. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
177. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
178. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
179. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
180. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
181. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
182. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
183. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
184. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
185. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
186. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
187. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
188. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
189. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 255 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
190. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 255 0 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
191. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 255 0 255 255 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
192. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 255 255 0 255 510 255 0 255 255 0 0 × 1 × 0 × 1 × –4 × 1 × 0 × 1 × 0 × Convolution
193. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 255 510 255 255 0 0 255 255 0 0 0 0 255 255 0 0 Convolution
194. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 255 255 0 0 0 255 255 255 0 0 0 255 255 255 510 Convolution
195. Edge Detection
196. Edge Detection
197. Edge Detection
198. Edge Detection
199. Line Detection
200. Line Detection
201. 0 1 0 1 –4 1 0 1 0
202. 0 1 0 1 –4 1 0 1 0 Laplace Filter
203. Sobel Filter 1 0 –1 2 0 –2 1 0 –1 1 2 1 0 0 0 –1 –2 –1
204. 1 0 –1 2 0 –2 1 0 –1 Sobel Filter 1 2 1 0 0 0 –1 –2 –1 Gy = Gx = =
205. Sobel Filter G = √Gx 2 + Gy 2
206. –1 0 1 –2 0 2 –1 0 1
207. –1 0 1 –2 0 2 –1 0 1
208. –1 0 1 –2 0 2 –1 0 1
211. Hough Transform
221. y
222. y
223. y r
224. y r θ
225. y r θ r = x cos(θ) + y sin (θ)
226. y
227. θ ~r ~r ~r 0 12 9 3 15 13 11 5 30 13 11 7 45 13 11 8 60 11 11 9 75 9 9 9 90 6 7 9 105 3 4 8 120 0 2 6 135 -3 0 4 150 -6 -3 2 165 -9 -6 0 y
228. θ ~r ~r ~r 0 12 9 3 15 13 11 5 30 13 11 7 45 13 11 8 60 11 11 9 75 9 9 9 90 6 7 9 105 3 4 8 120 0 2 6 135 -3 0 4 150 -6 -3 2 165 -9 -6 0 y
229. θ ~r ~r ~r 0 12 9 3 15 13 11 5 30 13 11 7 45 13 11 8 60 11 11 9 75 9 9 9 90 6 7 9 105 3 4 8 120 0 2 6 135 -3 0 4 150 -6 -3 2 165 -9 -6 0
230. θ ~r ~r ~r 0 12 9 3 15 13 11 5 30 13 11 7 45 13 11 8 60 11 11 9 75 9 9 9 90 6 7 9 105 3 4 8 120 0 2 6 135 -3 0 4 150 -6 -3 2 165 -9 -6 0
233. r θ
234. r θ
237. Performance
238. Every Instruction Matters
239. Avoid Copying
240. Avoid Copying ByteArray
241. Avoid Copying ByteArray ByteArray<ByteArray>
242. class ByteGrid( private val data: ByteArray, private val width: Int, private val height: Int )x
243. class ByteGrid( private val data: ByteArray, private val width: Int, private val height: Int )x{ fun get(x: Int, y: Int): Byte }x
244. class ByteGrid( private val data: ByteArray, private val width: Int, private val height: Int )x{ fun get(x: Int, y: Int): Byte fun rotate(rotation: Int) }x
245. class ByteGrid( private val data: ByteArray, private val width: Int, private val height: Int )x{ fun get(x: Int, y: Int): Byte fun rotate(rotation: Int) fun crop(x: Int, y: Int, width: Int, height: Int) }x
246. Avoid Allocating
247. fun doWork(image: ByteGrid): ByteGrid
248. fun doWork(image: ByteGrid, result: ByteGrid)
249. Optimize Kernels
250. val KERNEL = arrayOf( arrayOf(0, 1, 0), arrayOf(1, -4, 1), arrayOf(0, 1, 0) ) var sum = 0 for (kernelY in 0 until KERNEL.size) { for (kernelX in 0 until KERNEL[kernelY].size) { sum += KERNEL[kernelY][kernelX] * image.get(x, y) } }
251. !// Kernel: !// 0 1 0 !// 1 -4 1 !// 0 1 0 var sum = (image.get(x, y - 1) + image.get(x - 1, y) - 4 * image.get(x, y) + image.get(x + 1, y) + image.get(x, y + 1))
252. Performance
253. Performance • Kotlin inline functions
254. Performance • Kotlin inline functions • Be lazy
255. Performance • Kotlin inline functions • Be lazy • Native code
256. Performance • Kotlin inline functions • Be lazy • Native code • GPU
257. Quick Iteration
258. Modules!
259. project app
260. project app camera
261. project app camera camera-sample
262. Unit Tests
263. Unit Tests
264. Unit Tests
265. Unit Tests
267. github.com/mattprecious/snap
268. Matt Precious @mattprec SNAP SNAP SNAP SNAP Building Camera Features in a