顔写真メイクアップアプリの開発におけるプライバシー保護とコスト削減のための手法

iOSDC Japan 2023

https://fortee.jp/iosdc-japan-2023/proposal/6aa80fd7-ec73-4261-8ea0-563cbfa52df5

Transcript

1. akkey / AKIO ITAYA ϓϥΠόγʔอޢͱίετ࡟ݮͷͨΊͷख๏ AppBrew, Inc. إࣸਅϝΠΫΞοϓΞϓϦͷ։ൃʹ͓͚Δ
2. Խহ඼ߪೖͷ͝ܦݧ͸ʁ
3. Խহ඼ͷ৭બͼ͸೉͍͠ ஶ࡞ऀɿFreepik
 https://jp.freepik.com/free-photo/red-lipsticks-arrangement- fl at-lay_14604143.htm
4. όʔνϟϧϝΠΫʹ௅ઓ ஶ࡞ऀɿFreepik
 https://jp.freepik.com/free-photo/medium-shot-girl-taking-sel fi e_42085215.htm ʮόʔνϟϧϝΠΫςελʔʯ͸גࣜձࣾࢿੜಊͷొ࿥঎ඪͰ͢
5. ৭Λ બ୒ όʔνϟϧϝΠΫʹ௅ઓ
6. ৭Λ બ୒ 📸 ࡱӨ όʔνϟϧϝΠΫʹ௅ઓ
7. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ όʔνϟϧϝΠΫʹ௅ઓ
8. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू όʔνϟϧϝΠΫʹ௅ઓ
9. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू όʔνϟϧϝΠΫʹ௅ઓ
10. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू όʔνϟϧϝΠΫʹ௅ઓ ϩ ʔ Χ ϧ ׬ ݁
11. ݁࿦ 04
12. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू ϩ ʔ Χ ϧ ׬ ݁ إͷ෼ੳɾՃ޻Λ୺຤಺Ͱߦ͏ ϓϥΠόγʔอޢ
13. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू ϩ ʔ Χ ϧ ׬ ݁ إͷ෼ੳɾՃ޻Λ୺຤಺Ͱߦ͏ ίετ࡟ݮ
14. ϓϥΠόγʔอޢͱίετ࡟ݮͷͨΊͷख๏ إࣸਅϝΠΫΞοϓΞϓϦͷ։ൃʹ͓͚Δ
15. ։ൃ
16. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू إͷ෼ੳ
17. Vision Framework Apply computer vision algorithms to perform a variety of tasks on input images and video. https://developer.apple.com/documentation/vision
18. ࣄલ४උ
19. Core ML models https://github.com/john-rocky/CoreML-Models
20. https://github.com/john-rocky/CoreML-Models#face-parsing
21. ϓϩδΣΫτʹऔΓࠐΉ
22. ϓϩδΣΫτʹऔΓࠐΉ
23. ϓϩδΣΫτʹऔΓࠐΉ final class ViewModel { typealias FaceParsing = face_parsing init() { let mlModel = try! FaceParsing().model } 
 }
24. Core ML Helpers https://github.com/hollance/CoreMLHelpers
25. ϓϩδΣΫτʹऔΓࠐΉ
26. ίʔσΟϯά 08
27. إͷ෼ੳ final class ViewModel { init() { let mlModel = try! FaceParsing().model } 
 } MLModel ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
28. إͷ෼ੳ final class ViewModel { init() { let mlModel = try! FaceParsing().model let coreMLModel = try! VNCoreMLModel(for: mlModel) } 
 } MLModel VNCoreMLModel ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
29. إͷ෼ੳ final class ViewModel { init() { let mlModel = try! FaceParsing().model let coreMLModel = try! VNCoreMLModel(for: mlModel) let request = VNCoreMLRequest(model: coreMLModel) request.imageCropAndScaleOption = .scaleFill } 
 } MLModel VNCoreMLModel VNCoreMLRequest ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
30. إͷ෼ੳ final class ViewModel { private let faceImage = CIImage(image: UIImage(named: “face")!)! init() { 
 … } 
 } CIImage ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
31. إͷ෼ੳ final class ViewModel { private let faceImage = CIImage(image: UIImage(named: “face")!)! init() { 
 … let requestHandler = VNImageRequestHandler(ciImage: faceImage) } 
 } CIImage VNImageRequestHandler ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
32. إͷ෼ੳ final class ViewModel { private let faceImage = CIImage(image: UIImage(named: “face")!)! init() { 
 … let requestHandler = VNImageRequestHandler(ciImage: faceImage) try! requestHandler.perform([request]) } 
 } CIImage VNImageRequestHandler VNCoreMLRequest ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
33. VNCoreMLRequest إͷ෼ੳ final class ViewModel { private let faceImage = CIImage(image: UIImage(named: "face")!)! init() { … let requestHandler = VNImageRequestHandler(ciImage: faceImage) try! requestHandler.perform([request]) let result = request.results!.first as! VNCoreMLFeatureValueObservation } 
 } VNCoreMLFeatureValueObservation ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
34. إͷ෼ੳ final class ViewModel { private let faceImage = CIImage(image: UIImage(named: "face")!)! init() { … let requestHandler = VNImageRequestHandler(ciImage: faceImage) try! requestHandler.perform([request]) let result = request.results!.first as! VNCoreMLFeatureValueObservation let multiArray = result.featureValue.multiArrayValue! } 
 } MLMultiArray ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ VNCoreMLRequest VNCoreMLFeatureValueObservation
35. إͷ෼ੳ final class ViewModel { private let multiArray: MLMultiArray private let faceImage = CIImage(image: UIImage(named: "face")!)! init() { … let requestHandler = VNImageRequestHandler(ciImage: faceImage) try! requestHandler.perform([request]) let result = request.results!.first as! VNCoreMLFeatureValueObservation multiArray = result.featureValue.multiArrayValue! } 
 } ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ MLMultiArray VNCoreMLRequest VNCoreMLFeatureValueObservation
36. ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू 🔨 ฤू 10
37. Ϧοϓͷදݱ💄
38. Ϧοϓͷදݱ Normal
39. Ϧοϓͷදݱ Normal Overlay
40. Ϧοϓͷදݱ Normal Overlay Color Change
41. Ϧοϓͷදݱ Color Change Overlay Real
42. ੒෼Ͱߟ͑ͯΈΔ
43. إྉͱછྉ https://www.toyo-color.com/ja/products/organic_pigments/about.html إྉ છྉ
44. إྉͱછྉ https://www.toyo-color.com/ja/products/organic_pigments/about.html إྉ છྉ
45. https://lipscosme.com/products/337039
46. https://lipscosme.com/products/337039ʛhttps://ameblo.jp/rik01194/entry-12309635144.html إྉ છྉ
47. ৶ͷ৭Λิਖ਼͢Δͱྑͦ͞͏ʁ ※ ঺հࣄྫ͸͋͘·Ͱ΋ҰྫͰ͢
48. إྉͱછྉ https://www.toyo-color.com/ja/products/organic_pigments/about.html إྉ છྉ
49. CIColorCrossPolynomial The properties you use to con fi gure a color cross-polynomial fi lter https://developer.apple.com/documentation/coreimage/cicolorcrosspolynomial 14
50. CIColorCrossPolynomial https://note.com/yogox/n/n7aa72d268144#aXmbf ɹ: ܎਺
 ɹ: ೖྗըૉͷ੺੒෼஋
 ɹ: ೖྗըૉͷ྘੒෼஋
 ɹ: ೖྗըૉͷ੨੒෼஋ f(r,g,b)=k_0r+k_1g+k_2b+k_3r^2+k_4g^2+k_5b^2+k_6rg+k_7gb+k_8br+k_9
 https://sundryst.com/convenienttool_math/drawexpression.html ग़ྗըૉͷ֤৭੒෼஋
51. CIColorCrossPolynomial let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2]
52. CIColorCrossPolynomial let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2] let redVector = CIVector(values: redArray, count: redArray.count)
53. CIColorCrossPolynomial let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2] let redVector = CIVector(values: redArray, count: redArray.count) let editedCIImage = CIFilter( name: "CIColorCrossPolynomial", parameters: ["inputRedCoefficients": redVector] )!.outputImage! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
54. CIColorCrossPolynomial let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2] let redVector = CIVector(values: redArray, count: redArray.count) 
 let image = CIImage(image: UIImage(named: "color_test")!)! let editedCIImage = CIFilter( name: "CIColorCrossPolynomial", parameters: [ kCIInputImageKey: ciImage, "inputRedCoefficients": redVector ] )!.outputImage! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
55. CIColorCrossPolynomial let redArray: [CGFloat] = [2, 2, 2, 0, 0, 0, 0, 0, 0, 0.2] let redVector = CIVector(values: redArray, count: redArray.count) 
 let image = CIImage(image: UIImage(named: "color_test")!)! let editedCIImage = CIFilter( name: "CIColorCrossPolynomial", parameters: [ kCIInputImageKey: ciImage, "inputRedCoefficients": redVector ] )!.outputImage! let context = CIContext() let cgImage = context.createCGImage(editedCIImage, from: editedCIImage.extent)! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢
56. όʔνϟϧϝΠΫʹ௅ઓ ৭Λ બ୒ 📸 ࡱӨ إͷ ෼ੳ 🔨 ฤू 17
57. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ όʔνϟϧϝΠΫʹ௅ઓ
58. ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ όʔνϟϧϝΠΫʹ௅ઓ let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage! 
 
 let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)!
59. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage! 
 
 let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)! 
 let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size) ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ όʔνϟϧϝΠΫʹ௅ઓ
60. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage! 
 
 let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)! 
 let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size) 
 
 let compositedUpperLipImage = CIFilter( name: "CIBlendWithMask", parameters: [ kCIInputImageKey: editedCIImage, kCIInputBackgroundImageKey: faceImage, kCIInputMaskImageKey: upperLipCIImage ] )!.outputImage! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ όʔνϟϧϝΠΫʹ௅ઓ
61. let editedCIImage = CIFilter(name: "CIColorCrossPolynomial",…)!.outputImage! 
 
 let upperLipCGImage = multiArray.cgImage(…, outputType: FaceType.upperLip)! 
 let upperLipCIImage = CIImage(cgImage: …).resize(as: faceImage.extent.size) 
 
 let compositedUpperLipImage = CIFilter( name: "CIBlendWithMask", parameters: [ kCIInputImageKey: editedCIImage, kCIInputBackgroundImageKey: faceImage, kCIInputMaskImageKey: upperLipCIImage ] )!.outputImage! 
 
 let context = CIContext() let cgImage = context.createCGImage( 
 compositedUpperLipImage, 
 from: compositedUpperLipImage.extent 
 )! ίʔυ؆ུԽͷͨΊʹڧ੍ΞϯϥοϓΛଟ༻͍ͯ͠·͢ όʔνϟϧϝΠΫʹ௅ઓ
62. Sample https://github.com/AkkeyLab/vision-makeup
63. ࣗݾ঺հ
64. akkey ஥ؒืूத डߨੜืूத
65. Thank you !!