Face Recognition with Vision & Core ML

Transcript

1. Face Recognition on mobile applications Deep Learning & Augmented Reality
2. Integration inside a Mobile App
5. Tell Me Why Diving Deeper Performances Privacy
6. Machine Learning Machine Learning on Mobile
7. Machine Learning Machine Learning on Mobile
8. Machine Learning Machine Learning on Mobile
9. 2 Main Solutions 2 Main Solutions
10. Google: TensorFlow 2 Main Solutions
11. Apple: Vision & Core ML 2 Main Solutions
12. Deep Learning for Face Recognition
13. Two Components
14. Machine Learning
15. Vision ▼ Face and face landmark detection ▼ Text detection ▼ Barcode recognition ▼ Feature tracking ▼ Custom Core ML models
16. Face Rectangle Request let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right, options: [:]) let request = VNDetectFaceRectanglesRequest() try imageRequestHandler.perform([request]) guard let results = request.results as? [VNFaceObservation] else { return } for result in results { result.confidence // 0...1 result.landmarks // Face Landmarks result.boundingBox // Rectangle Around the Face }
17. Machine Learning
18. Apple: Core ML Diving Deeper
19. Core ML: Execution on CPU or GPU? Diving Deeper ▼ CPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework
20. Core ML: Execution on GPU or GPU? Diving Deeper ▼ GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU
21. Core ML: Execution on GPU or GPU? Diving Deeper ▼ GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU ▼ iOS chooses for You /!\
22. Apple: Core ML Diving Deeper
23. Apple: Core ML Diving Deeper
25. Apple: Core ML Diving Deeper
26. Diving Deeper Main Hurdles ▼ Conversion ▼ Performances ▼ Model Size
27. Apple: Core ML Diving Deeper
28. Core ML Tools Diving Deeper https://github.com/apple/coremltools ▼ Convert ▼ Predict (Test)
29. Core ML Tools Diving Deeper from inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output" ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
30. Diving Deeper Easier Said Than Done ▼ Input Normalization ▼ Custom Layers
31. Custom Layers // ... def scaling(x, scale): return x * scale // ...
32. Core ML Tools: Input Normalization and Conversion Diving Deeper from inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output", add_custom_layers=True, image_scale=2/255.0, red_bias=-1, green_bias=-1, blue_bias=-1, custom_conversion_functions={ "Lambda": convert_lambda } ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
33. Core ML+ Xcode Diving Deeper
34. Core ML in Swift Diving Deeper let model = try VNCoreMLModel(for: facenet_keras_weights_coreml().model) let coreMLRequest = VNCoreMLRequest(model: model) // Options coreMLRequest.imageCropAndScaleOption = .scaleFit // ...
35. Diving Deeper Easier Said Than Done ▼ Input Normalization ▼ Custom Layers ▼ Input Normalization ▼ Custom Layers ▽ Performance
36. Custom Layers // ... def scaling(x, scale): return x * scale // ...
37. Custom Layers: No Optimization @objc(scaling) class Scaling: NSObject, MLCustomLayer { // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) for j in 0..<input.count { let x = input[j].doubleValue let y = x * scale output[j] = NSNumber(value: y) } } } }
38. Custom Layers: No Optimization
39. Custom Layers: Using Accelerate.framework @objc(scaling) class Scaling: NSObject, MLCustomLayer { // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { var scale = Float(self.scale) for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) let count = input.count let inputPointer = UnsafeMutablePointer<Float>(OpaquePointer(input.dataPointer)) let outputPointer = UnsafeMutablePointer<Float>(OpaquePointer(output.dataPointer)) vDSP_vsmul(inputPointer, 1, &scale, outputPointer, 1, vDSP_Length(count)) } } }
40. Custom Layers: Using Accelerate.framework
41. Custom Layers: Using Metal Shader (GPU) #include <metal_stdlib> using namespace metal; kernel void scaling( texture2d_array<half, access::read> inTexture [[texture(0)]], texture2d_array<half, access::write> outTexture [[texture(1)]], constant float& scale [[buffer(0)]], ushort3 gid [[thread_position_in_grid]]) { if (gid.x >= outTexture.get_width() || gid.y >= outTexture.get_height()) { return; } const float4 x = float4(inTexture.read(gid.xy, gid.z)); const float4 y = x * scale; outTexture.write(half4(y), gid.xy, gid.z); }
42. Custom Layers: Using Metal Shader (GPU) @objc(scaling) class Scaling: NSObject, MLCustomLayer { if let encoder = commandBuffer.makeComputeCommandEncoder() { for i in 0..<inputs.count { encoder.setTexture(inputs[i], index: 0) encoder.setTexture(outputs[i], index: 1) var scale = self.scale encoder.setBytes(&scale, length: MemoryLayout<Float>.size, index: 0) encoder.dispatch(pipeline: scalingPipeline, texture: inputs[i]) encoder.endEncoding() } } }
43. Custom Layers: Using Metal Shader (GPU)
44. Two Components
46. Augmented Reality
47. Augmented Reality
48. Augmented Reality
49. Augmented Reality = +
50. Augmented Reality Vision Framework
51. Augmented Reality
52. Adding 3D Objects: What’s a Node? SCNNode()
53. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) { let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return }
54. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) { let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point
55. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) { let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point let constraint = SCNBillboardConstraint() baseNode.constraints = [constraint] arSceneView.scene.rootNode.addChildNode(pointerNode) }
56. What’s Next
57. Summing Up What’s Next ▼ Optimizing Model Size ▽ Quantization ▼ Better Performances ▼ Core ML 2 (Beta) ▼ Create ML 2.0
58. Summing Up
59. Summing Up Recap ▼ What is DL ▼ How to use DL for Face Recognition ▼ How to import a face recognition model inside a mobile app ▼ How to make use of a ML model to create an AR experience on a modern phone
60. References
61. References References ▼ http://machinethink.net/blog/ ▼ https://developer.apple.com/wwdc/ REF