Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

配信アプリのためのリアルタイムプッシュ通知ぼかしの夢

naru-jpn
September 02, 2023

 配信アプリのためのリアルタイムプッシュ通知ぼかしの夢

naru-jpn

September 02, 2023
Tweet

More Decks by naru-jpn

Other Decks in Programming

Transcript

  1. override func processSampleBuffer( _ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType )

    { switch sampleBufferType { case RPSampleBufferType.video: break case RPSampleBufferType.audioApp: break case RPSampleBufferType.audioMic: break @unknown default: fatalError("Unknown type of sample buffer") } } 4BNQMF)BOEMFSTXJGU
  2. override func processSampleBuffer( _ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType )

    { switch sampleBufferType { case RPSampleBufferType.video: break case RPSampleBufferType.audioApp: break case RPSampleBufferType.audioMic: break @unknown default: fatalError("Unknown type of sample buffer") } } 4BNQMF)BOEMFSTXJGU
  3. override func processSampleBuffer( _ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType )

    { switch sampleBufferType { case RPSampleBufferType.video: break ॲཧ࣌ؒͱ࢖༻ϝϞϦͷ੍໿ ͜ͷؔ਺͸ܧଓతʹݺ͹Εɺಈը΍Ի੠ʹରԠ͢ΔTBNQMF#VGGFS͕ڙڅ͞ΕΔ ಈը͕'14ͱ͢Δͱɺඵؒʹճݺ͹ΕΔʢ໿NTִؒʣ ϝϞϦ࢖༻ྔ͸؂ࢹ͞Ε͍ͯͯɺ.#Λ௒͑Δͱϓϩηε͕ऴྃ͢Δ
  4. case RPSampleBufferType.video: let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)! let pixelFormatType = CVPixelBufferGetPixelFormatType(imageBuffer)

    // pixelFormatType == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange TBNQMF#VGGFS͔ΒಈըͷϑϨʔϜʹ૬౰͢ΔJNBHF#VGGFSΛऔಘͯ͠ ϑΥʔϚοτΛௐ΂Δ L$71JYFM'PSNBU5ZQF@:Q$C$S#J1MBOBS'VMM3BOHF
  5. L$71JYFM'PSNBU5ZQF@"3(#͸ 6*OUͷ஋͕" 3 ( #ͷॱʹฒΜͰ͍Δ 6*OU [ … A0, R0,

    G0, B0, A1, R1, G1, B1, A2, R2, G2, B2, … A3, R3, G3, B3, A4, R4, G4, B4, A5, R5, G5, B5, … A6, R6, G6, B6, A7, R7, G7, B7, A8, R8, G8, B8, … … ] 0 1 2 3 4 5 6 7 8
  6. // (0,0) Λࢦ͢ΞυϨε let address = CVPixelBufferGetBaseAddress(imageBuffer) // ઌ಄ͷϐΫηϧ͔Β ARGB

    ͷॱͰ৭৘ใ͕֨ೲ͞Ε͍ͯΔ let arbg = address!.assumingMemoryBound(to: UInt8.self) let a_0_0 = arbg[0] // A let r_0_0 = arbg[1] // R let g_0_0 = arbg[2] // G let b_0_0 = arbg[3] // B let a_1_0 = arbg[4] // A L$71JYFM'PSNBU5ZQF@"3(#ܗࣜͷ ը૾σʔλͷઌ಄ΞυϨεΛऔಘͯ͠ɺըૉʹΞΫηε͢Δ
  7. ً౓ : ฏ໘͸ϐΫηϧͷً౓ʹରԠ͢Δ 6*OUͷ஋ d ͕ॱʹฒΜͰ͍Δ 6*OU [… Y0, Y1,

    Y2, Y3,… Y4, Y5, Y6, Y7,… Y8, Y9, Y10, Y11,… Y12, Y13, Y14, Y15,… …] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  8. ৭ࠩ $C$S ฏ໘͸6*OUͷ஋ d ͕$C $SͷॱʹฒΜͰ͍Δ 0 1 2 3

    6*OU [… Cb0, Cr0, Cb1, Cr1,… Cb2, Cr2, Cb3, Cr3,… …]
  9. ً౓ : ฏ໘ ৭ࠩ $C$S ฏ໘ 0 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 L$71JYFM'PSNBU5ZQF@:Q$C$S#J1MBOBS'VMM3BOHFͰ͸ ৭ࠩ $C$S ฏ໘ͷղ૾౓͸ɺً౓ : ฏ໘ͷ൒෼Ͱ͋Δ ը໘දࣔ
  10. ը໘දࣔ 0 1 2 3 4 5 6 7 8

    9 10 11 12 13 14 15 ً౓ : ฏ໘ ৭ࠩ $C$S ฏ໘ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 ը໘දࣔ : $C $S
  11. ը໘දࣔ 0 1 2 3 4 5 6 7 8

    9 10 11 12 13 14 15 ً౓ : ฏ໘ ৭ࠩ $C$S ฏ໘ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 ը໘දࣔ : $C $S
  12. ը໘දࣔ 0 1 2 3 4 5 6 7 8

    9 10 11 12 13 14 15 ً౓ : ฏ໘ ৭ࠩ $C$S ฏ໘ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 ը໘දࣔ : $C $S
  13. let length = 1_000_000 let v1 = UnsafeMutablePointer<Float>.allocate(capacity: length) let

    v2 = UnsafeMutablePointer<Float>.allocate(capacity: length) v1.initialize(repeating: 10, count: length) v2.initialize(repeating: 20, count: length) ௕͞ສͷߦྻʹର͢Δॲཧ଎౓Λൺֱ͢Δ
  14. // Others for i in 0..<length { out[i] = v1[i]

    + v2[i] } // Accelerate vDSP_vadd(v1, 1, v2, 1, out, 1, vDSP_Length(length)) 0UIFST "DDFMFSBUF <NT> <NT> ߦྻͷཁૉͷ଍͠ࢉ
  15. // Others for i in 0..<length { out[i] = v1[i]

    * v2[i] } // Accelerate vDSP_vmul(v1, 1, v2, 1, out, 1, vDSP_Length(length)) 0UIFST "DDFMFSBUF <NT> <NT> ߦྻͷཁૉͷֻ͚ࢉ
  16. ۮ਺ΠϯσοΫε     ͷཁૉʹͷΈΛ଍͢ // Others (whole loop)

    for i in 0..<length { if i % 2 == 0 { v[i] = v[i] + 1 } } // Others (loop with stride) for i in stride(from: 0, to: length, by: 2) { v[i] = v[i] + 1 } // Accelerate var value: Float = 1 vDSP_vsadd(v, 2, &value, v, 2, vDSP_Length(length / 2)) 0UIFST XIPMFMPPQ 0UIFST MPPQXJUITUSJEF "DDFMFSBUF <NT> <NT> <NT>
  17. ΑΓେ͖͍TUSJEFΛࢦఆͯ͠ ҰఆͷִؒΛۭ͚ͨϕΫτϧ্ͷܭࢉ͕Ͱ͖Δ // Accelerate var value: Float = 1 vDSP_vsadd(v,

    2, &value, v, 2, vDSP_Length(length / 2)) // Others (loop with stride) for i in stride(from: 0, to: length, by: 2) { v[i] = v[i] + 1 } " 3 # (΍$C $SͷΑ͏ʹ ౳ִؒͰಉ͡छྨͷ৘ใ͕ฒΜͰ͍Δ৔߹ͳͲʹศར
  18. W*NBHF https://developer.apple.com/documentation/accelerate/vimage vImage is particularly suited for: • Ef fi

    ciently processing large images • Real-time video processing software • Scienti fi c applications that require high-accuracy numerical calculations ެࣜΑΓҾ༻
  19. W*NBHF $71JYF#VGGFS QMBOF let data = UnsafeMutableRawPointer(addressOfPlane) let buffer =

    vImage_Buffer( data: data, height: height, width: width, rowBytes: rowBytes ) W*NBHF@#VGGFS
  20. ୺຤ͷը໘αΠζ͝ͱʹϓογϡ௨஌ͷࠨӈͷϚʔδϯΛܭଌͯ͠ ݕग़ͷࡍͷܭࢉʹ࢖༻͍ͯ͠Δ … // iPhone 12,13,14 iPhone Pro 12,13 "1170x2532":

    .init(width: 1170, height: 2532, x: 24), "2532x1170": .init(width: 2532, height: 1170, x: 432), // iPhone Pro Max 14 "1290x2796": .init(width: 1290, height: 2796, x: 24), "2796x1290": .init(width: 2796, height: 1290, x: 564), // iPad 3rd,4th,5th,6th / iPad mini 2nd,3rd,4th,5th / iPad Air 1st,2nd "1536x2048": .init(width: 1536, height: 2048, x: 212), "2048x1536": .init(width: 2048, height: 1536, x: 468), …
  21. ৭ ً౓ ৭ࠩ ͷมԽ : $C $S : $C $S

    c::c c$C$Cc c$S$Sc
  22. ৭ ً౓ ৭ࠩ ͷมԽ : $C $S c::c c$C$Cc c$S$Sc

    : $C $S ˞Ҏ߱Ͱ͸ɺ؆୯ͷͨΊʹً౓ͷΈͷܭࢉͱݟཱͯͯߟ͑·͢
  23. W W ڥքˠ৭ͷมԽ͕େ͖͍ vDSP_vsub(v1, 1, v2, 1, boundary, 1, length)

    vDSP_vabs(boundary, 1, boundary, 1, length) CPVOEBSZcWWc ڥքͷϕΫτϧͷཁૉͷࠩͷઈର஋͸ɺҰఆΑΓେ͖͍ CPVOEBSZ<J>DPOTU
  24. W W ಺ଆͷྖҬˠ৭ͷมԽ͕খ͍͞ vDSP_vsub(v2, 1, v3, 1, inner, 1, length)

    vDSP_vabs(inner, 1, inner, 1, length) JOOFScWWc ಺ଆͷϕΫτϧͷཁૉͷࠩͷઈର஋͸ɺҰఆΑΓখ͍͞ JOOFS<J>DPOTU
  25. JOOFS ಺ଆͷྖҬʹରͯ͠ɺڥքͷํ͕৭ͷมԽ͕େ͖͍ vDSP_vdiv(boundary, 1, inner, 1, out, 1, length) PVU<J>CPVOEBSZ<J>JOOFS<J>

    ϓογϡ௨஌ͷଆ໘Ͱ͸ɺ৭ͷมԽͷൺ͕ҰఆΑΓେ͖͍ CPVOEBSZ PVU<J>CPVOEBSZ<J>JOOFS<J>DPOTU େ খ
  26. // ڥք vDSP_vsub(v1, 1, v2, 1, boundary, 1, length) vDSP_vabs(boundary,

    1, boundary, 1, length) // ಺ଆ vDSP_vsub(v2, 1, v3, 1, inner, 1, length) vDSP_vabs(inner, 1, inner, 1, length) // ڥք / ಺ଆ vDSP_vdiv(boundary, 1, inner, 1, out, 1, length) ϓογϡ௨஌͕௨ա͢Δ۠ؒʹ౉ͬͯܭࢉΛ͢Δ
  27. [100, 100, 100, 10, 10, 10, 10, …] [100, 100,

    100, 10, 10, 10, 10, …] ϕΫτϧΛͣΒͯ͠Ҿ͖ࢉ͢ΔͱɺมԽ͕Θ͔Δ
  28. [100, 100, 100, 10, 10, 10, 10, …] [100, 100,

    100, 10, 10, 10, 10, …] - [ 0, 0, 90, 0, 0, 0, …] ϕΫτϧΛͣΒͯ͠Ҿ͖ࢉ͢ΔͱɺมԽ͕Θ͔Δ
  29. BEWBODFEͰͣΒͨ͠ϕΫτϧͷࠩΛܭࢉ͠ɺ৭ͷมԽ఺Λ୳Δ vDSP_vsub(v, 1, v.advanced(by: 1), 1, out, 1, length-1) vDSP_vabs(out,

    1, out, 1, length) PVUcWWBEWBODFE  c PVU<J>͕ҰఆΑΓେ͖͍৔ॴ͕ϓογϡ௨஌ͷఈͷީิʹͳΔ W WBEWBODFE 
  30. // ً౓(Y)ฏ໘ͷઌ಄Λࢦ͢ΞυϨε let address0 = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0) // ৭ࠩ(CbCr)ฏ໘ͷઌ಄Λࢦ͢ΞυϨε let

    address1 = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1) ً౓ : ฏ໘ͱ৭ࠩ $C$S ฏ໘͸ผʑʹՃ޻͢Δඞཁ͕͋Δ
  31. ً౓ : ฏ໘͸ϐΫηϧͷً౓ʹରԠ͢Δ 6*OUͷ஋ d ͕ॱʹฒΜͰ͍Δ 6*OU [… Y0, Y1,

    Y2, Y3,… Y4, Y5, Y6, Y7,… Y8, Y9, Y10, Y11,… Y12, Y13, Y14, Y15,… …] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15