iPhoneでコスプレをする技術

iPhoneでコスプレをする技術

俺コン 2018 Summer
#orecon_ios

Transcript

1. iPhoneͰίεϓϨΛ͢Δٕज़ ాதୡ໵ (@tattn) Զίϯ 2018 Summer #orecon_ios
2. ాத ୡ໵ (@tattn) • Yahoo!৐׵Ҋ಺ • iOSΞϓϦΤϯδχΞ @tattn @tanakasan2525 @tattn
3. ࡢ೔ͷApple Event؍·͔ͨ͠ʁ
4. iOS12ͷ঺հϖʔδ IUUQTXXXBQQMFDPNKQJPTJPT
5. ৐׵Ҋ಺͕ϖʔδʹࡌΓ·ͨ͠ IUUQTXXXBQQMFDPNKQJPTJPT
6. iPhoneͰίεϓϨΛ͢Δٕज़ IUUQTGPSUFFKQJPTEDKBQBOQSPQPTBMECGCCGCBBGCDGF
7. όʔνϟϧYouTuber
 ஌͍ͬͯ·͔͢ʁ
8. όʔνϟϧYouTuber (VTuber) ஌Βͳ͍ਓ͸ϠϑʔͰݕࡧʂʮVTuberʯ YouTubeͳͲͷಈը഑৴αʔϏε্Ͱ
 2D΍3DͷΞόλʔͰ഑৴͍ͯ͠Δਓͷ͜ͱɻ
 ϦΞϧͱόʔνϟϧͷମΛಉظ͢Δٕज़Λ࢖͍ͬͯΔ͜ͱ͕ଟ͍ɻ
9. VTuberΛࢧ͑Δٕज़ • ମΛಈ͔͢ • खಈɺKinectɺVRػثɺϞʔγϣϯΩϟϓνϟʔػث • ද৘Λಈ͔͢ • खಈɺFaceRigɺDlibɺVisionɺARKit • (੠Λม׵͢Δ) • ιϑτ΢ΣΞ (࿀੠ͳͲ)ɺϋʔυ΢ΣΞ (VT-3ͳͲ) • දࣔ͢Δ • UnityɺFaceRigɺSceneKit (ҰྫͰ͢)
10. ٕज़తʹ΋໘ന͘ɺ৭ʑ࡞ͬͯΈ·ͨ͠ ※͜Ε͸Oculus RiftͱUnityΛ࢖͍ͬͯ·͢ VRδΣϯΨ VRΩϟονϘʔϧ VTuberͬΆ͍ࢹ఺ ©Kizuna AI IUUQTUXJUUFSDPNUBOBLBTBOTUBUVT
11. εϚϗͰ΋VTuberʹͳΕΔ
12. ͦ͏ɺiPhoneͳΒͶɻ
13. ͜ΜͳΞϓϦΛ࡞Γ·ͨ͠ Α͔ͬͨΒ༡ΜͰΈͯͶˠ iPhone X × ARKit × SceneKit CoreAnimation × Accelerate ʮVTuberʯͰݕࡧ
14. ࢖༻ٕज़ Face Tracking ϨϯμϦϯάɾϞʔγϣϯ TrueDepthΧϝϥ ܭࢉ iPhone X × ARKit × SceneKit CoreAnimation × Accelerate
15. ARKit ARFaceAnchorͰإͷ࢟੎ΛऔಘΛऔಘ
 BlendShapeLocationͰإͷύʔπͷঢ়ଶΛऔಘ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBSLJUBSGBDFBODIPS
16. Face Trackingͷ࣮ߦ guard ARFaceTrackingConfiguration.isSupported else { return } let configuration = ARFaceTrackingConfiguration() configuration.isLightEstimationEnabled = true let arSession = ARSession() arSession.delegate = self arSession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
17. إͷύʔπͷঢ়ଶऔಘ extension ViewController: ARSessionDelegate { func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy .compactMap({ $0 as? ARFaceAnchor }) .first else { return } if let mouthOpenness = faceAnchor.blendShapes[.jawOpen]?.doubleValue { let e = 0.05 // ޱΛด͡΍͘͢͢Δ let mouthOpennessX8 = mouthOpenness * 8 // ޱΛ։͚΍͘͢͢Δ let value = mouthOpennessX8 < e ? 0 : mouthOpennessX8 > 1 - e ? 1 : mouthOpennessX8 // Ϟσϧͷߋ৽ }
18. SceneKit SCNSceneʹΩϟϥΫλʔΛ഑ஔ͠
 SCNViewͰSceneΛදࣔ ©Kizuna AI
19. ϞσϧͷಡΈࠐΈ ϑϦʔͷϞσϧ͕ଟ͘ɺ
 ද৘ͷϞʔϑΟϯά΍ϞʔγϣϯͳͲͷ
 ޓ׵ੑ͕ߴ͍MMDϞσϧΛࠓճ͸࢖༻ ※MMD=MikuMikuDance IUUQEJDOJDPWJEFPKQBNJLVNJLVEBODF
20. MMDͷಡΈࠐΈ ϑΥʔϚοτͷ࢓༷ࣗମ͸ެ։͞Ε͍ͯΔ IUUQLLILTFFTBBOFUDBUFHPSZIUNM ↑zipͷதʹ࢓༷ॻ͕͋Δ IUUQTHJUIVCDPNNBHJDJFO..%4DFOF,JU ࠓճ͸MMDSceneKitΛ࢖༻
21. Ωϟϥͷύʔπͷऔಘ let model: SCNNode = try loadMMDModel(fromName: "kizunaai/kizunaai.pmx") let neck = model.childNode(withName: "ट", recursively: true) let leftEye = model.childNode(withName: "ࠨ໨", recursively: true) let rightEye = model.childNode(withName: "ӈ໨", recursively: true) MMDͷNode (Ϙʔϯ) ໊ʹ͸҉໧తͳϓϩτίϧ͕͋ΔͷͰ
 ผͷϞσϧ΋ͦͷ··Ͱେମಈ͘ 
 (ͨ·ʹશ֯ΧλΧφͱ൒֯ΧλΧφͷදهΏΕ͕...ţŜŖŪͳͲ)
22. CoreAnimation / SceneKit let keyPath = "morpher.weights.eye" var animation = CAKeyframeAnimation(keyPath: keyPath) animation.values = [AnyObject]() animation.keyTimes = [NSNumber]() animation.timingFunctions = [CAMediaTimingFunction]() ... let animationGroup = CAAnimationGroup() animationGroup.animations = [CAAnimation]() ... animationGroup.animations?.append(animation) ... let animation = SCNAnimation(caAnimation: animationGroup) let player = SCNAnimationPlayer(animation: animation) model.addAnimationPlayer(player, forKey: nil) player.play()
23. Accelerate.framework ύϑΥʔϚϯεʹಛԽͨ͠ࢉज़ԋࢉϑϨʔϜϫʔΫ ࠓճ͸ simd ͱ͍͏
 খ͞ͳϕΫτϧͱߦྻΛѻ͏ϞδϡʔϧΛར༻ ※ simd = Single Instruction Multiple Data
24. औಘͨ͠σʔλΛମͱटʹ൓ө͢Δ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard let faceAnchor = anchors .lazy.compactMap({ $0 as? ARFaceAnchor }).first else { return } let transform: simd_float4x4 = faceAnchor.transform let quaternion = simd_quaternion(transform) neck.simdWorldOrientation = quaternion let translate4: simd_float4 = transform.columns.3 let translate = simd_float3(translate4.x * 5, translate4.y * 5, translate4.z * 50 + 20) // w = 1 model.simdPosition = translate ࠨӈ͸͋·Γಈ͔ͣɺલޙ͸ಈ͖΍͘͢͢Δ ARKit/SceneKit͸ӈखܥ
25. ͜Ε·Ͱઆ໌ٕͨ͠ज़Λ૊Έ߹ΘͤΔͱ όʔνϟϧͳମΛಘΔ͜ͱ͕Ͱ͖·͢ʂ
26. VRM
27. VRMͱ͸ IUUQTEXBOHPHJUIVCJPWSN υϫϯΰ͕࡞ͬͨVR༻ͷ3DϞσϧϑΥʔϚοτ
 4݄ʹެ։͞ΕɺVRք۾Ͱ࿩୊ʹͳ͍ͬͯΔ ɾχίχཱମ
 ɾόʔνϟϧΩϟετ
 ɾcluster
 ɾVDraw
 ɾVΧπ ͳͲͰར༻͞Ε͍ͯΔ
28. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦΛ࡞Γ·ͨ͠ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit
29. VRMΛiOSͰಡΈࠐΊΔϥΠϒϥϦ IUUQTHJUIVCDPNUBUUO73.,JU VRMKit ϝλσʔλͷऔಘͱ
 Ϟσϧͷද͕ࣔՄೳ
30. ͊͞ɺΈͳ͞Μ΋όʔνϟϧͳੈք΁
31. ͋Γ͕ͱ͏͍͟͝·ͨ͠