Macとオーディオ再生 2024/11/02

macOS native Symposium #10 での発表資料です。

https://macos-native.github.io/backnumbers.html

Transcript

1. MacͱΦʔσΟΦ࠶ੜ 2024/11/02 Yusuke Ito Symposium #10 macOS
2. About me • Yusuke Ito • Web (TypeScript, React, Node.js) macOS ωΠςΟϒΞϓϦ (Minutes) iOS ωΠςΟϒΞϓϦ (App Store Best of 2014) • ϝΠϯ͸Web͕ͩɺΦʔσΟΦػثͷ։ൃɾ ੡଄+macOSΞϓϦ https://audio.current.directory/
4. Ի੠σʔλͷجຊ
5. PCM (LPCM) Linear Pulse Code Modulation Ϗοτਂ౓=ৼ෯ (16, 24bit) αϯϓϦϯάप೾਺(1ඵ͋ͨΓͷ෼ׂ਺) 44.1kHz(CD) 48kHz, 96kHz (Ի੍ָ࡞) Ի੠ͷੜσʔλ 0 -32768 32767
6. LPCMͷσʔλྔ • 44.1kHz (CD) αϯϓϦϯάप೾਺ • 16bit (CD) Ϗοτਂ౓ • 3෼ (180ඵ) • 44100 (sample/sec) × 2 (byte, 16bit) × 2 (channel, L R) × 180 (secs) • = 31,752,000 (bytes) ≈ 30 MB (megabytes)
7. ίϯςφͱίʔσοΫ ίϯςφ: QuickTime (.mov) MPEG-4 (.m4a, .m4v) Ի੠ AAC ө૾ H.264 ϝλσʔλ ΞʔςΟετ໊… ΞϧόϜ໊… ϝλσʔλ ίʔσοΫ: H.264 AAC Apple Lossless MP3, ATRAC (MD) JPEG PNG ϑΝΠϧͷ಺෦
8. ίʔσοΫͱσίʔυ Ի੠ AAC σίʔυ LPCM Ի੠ੜσʔλ Τϯίʔυ ίʔσοΫ
9. Mixing & Playback Ի੠σʔλͷྲྀΕ Χʔωϧ ϛΩαʔ σόΠε 🔊 App A App B γεςϜܯࠂԻ App C ഉଞϞʔυ OR (αϯϓϦϯάप೾਺ม׵) 96kHz 48kHz 44.1kHz 96kHz LPCM LPCM
10. ΦʔσΟΦAPIΛ࢖͏
11. Audio APIs on Mac • AVFoundation: ؆୯ʹ࢖͑ΔΫϥε܈ • AVAudioPlayerͳͲ, ϑΝΠϧ(M4A…)Ϩϕϧͷ΍ΓऔΓ • Audio Toolbox: • σίʔυɾΤϯίʔυɾαϯϓϦϯάप೾਺ม׵(SRC)ͳͲ • Ի੠σʔλ(PCM)Ϩϕϧͷ΍ΓऔΓ • Core Audio: ϋʔυ΢ΣΞͱͷ௨৴ • ϋʔυ΢ΣΞΛ௚઀ར༻͢Δ৔߹ Core Audio Hardware Audio Toolbox AVFoundation
12. Core Audio Λ࢖͏ ࠷௿ϨΠϠʔͷAPI • ΦʔσΟΦσόΠεʹग़ྗ͞ΕΔσʔλΛࣗ෼ͰίϯτϩʔϧͰ͖Δ • ඞཁͳλΠϛϯάͰίʔϧόοΫ͕ݺ͹ΕΔͷͰԻ੠σʔλ(PCMੜσʔλ)Λ ౉͢ • σʔλͷ४උ͕࠶ੜʹؒʹ߹͏Α͏ʹ͢Δ • ΠϯλʔϑΣʔε͸Cݴޠ • Audio Toolbox, Core Audio
13. ԻָPlayerΛ࡞Δ σʔλͷྲྀΕ αϯϓϦϯά प೾਺ม׵ (SRC) σίʔμʔ ϑΝΠϧ .m4a Apple Lossless όοϑΝ Core Audio 🔊 औΓʹ͍͘ Audio Toolbox Core Audio LPCM LPCM LPCM Apple Lossless App C ഉଞϞʔυ
14. ഉଞϞʔυ • API: Hog Mode • ΧʔωϧϛΩαʔΛհ͞ͳ͍ • →αϯϓϦϯάप೾਺ม׵ͳͲ͕ߦΘΕͳ͍ • →ଞͷΞϓϦέʔγϣϯͷԻ͸ग़ྗ͞Εͳ͍ • ΦʔσΟΦϋʔυ΢ΣΞʹ௚઀ΞΫηε • Ի੠(LPCM)σʔλΛૹͬͯ࠶ੜ
15. Swift & Concurrency
16. Swift & Realtime • ϦΞϧλΠϜॲཧͷอূ͸͞Ε͍ͯͳ͍(͸ͣ) • Swiftಛ༗ͷॲཧͷΦʔόʔϔου͕͋ΔՄೳੑ • ΫϦςΟΧϧͳͷ͸࠷ޙͷίʔϧόοΫՕॴͷΈ • ࣮ࡍ͸SwiftͰ໰୊ͳ͠ • ໰୊͋ΓͳΒCͰॻ͘ • ͦΕҎ֎ͷՕॴ͸ઈରϦΞϧλΠϜͰͳͯ͘΋໰୊ͳ͠
17. Concurrency • (Իָ࠶ੜPlayerͰ͸) ฒߦॲཧͷඞཁੑ͸ബ͍ • ͨͩ͠… • ಉ͡ΦϒδΣΫτΛڞ༗͢Δ • όοϑΝʔ(ϝϞϦɾϑΝΠϧ), ΦʔσΟΦσόΠε, σίʔμʔ • ಉ࣌ʹΞΫηε͠ͳ͍࢓૊Έ͸ඞཁ
18. ಉظॲཧɾഉଞॲཧ • 1ͭͷϦιʔε(ΦϒδΣΫτ)ʹಉ࣌ʹΞΫηε͠ͳ͍Α͏ʹ͢Δ • ಉ࣌ΞΫηεͷྫ • ಡΈࠐΈதʹॻ͖ࠐΉ • εϨουAͰಡΈࠐΈதʹεϨουBͰ΋ಡΈࠐΉ
19. ഉଞॲཧ • ର৅͕ಉظؔ਺ͷ৔߹ • NIOLock - swift-nio (All Platforms) • OSAllocatedUnfairLock (macOS 13~)
20. struct AudioSytemObject { let id: AudioObjectID = ... let lock = NIOLock() func get(addr: T) throws -> [T.DataType] { try lock.withLock { let memory = ... let statusData = AudioObjectGetPropertyData(id, &propAddr, 0, nil, &propSize, memory) return (0..<count).map { memory[$0] } } } func set(values: [T.DataType], forAddr addr: T) throws { try lock.withLockVoid { ... let status = AudioObjectSetPropertyData(id, &propAddr, 0, nil, UInt32(propSize), memory) } } } कΓ͍ͨΦϒδΣΫτ
21. όοϑΝ (ϝϞϦ) εϨουB εϨουA ॻ͖ࠐΈ ಡΈࠐΈ
22. final class PlaybackBuffer { var buffer = Data() let lock = NIOLock() func write(_ data: Data) { lock.withLockVoid { buffer.append(data) } } func read() -> Data? { return lock.withLock { if buffer.isEmpty { return nil } let buf = buffer buffer.removeAll() // clear data return buf } } }
23. Actor vs Lock • Actorͩͱisolated context (isolation boundary)Λ௒͑ΒΕͳ͍ • ݺͼग़͠ݩ͕CͷίʔϧόοΫ(ಉظ)ͷ৔߹ͳͲ • nonisolated ʹ͢Δ? →ࣗલͷഉଞॲཧ͕ඞཁ • await ͢Δ? →ಉظઐ༻ίʔϧόοΫͳͷͰෆՄ func AudioIOProc(buffer: PlaybackBuffer, outOutputData: UnsafeMutablePointer<AudioBufferList>) -> OSStatus { let data = buffer.read() memcpy(outOutputData, data, length) ...
24. ඇಉظؔ਺ͷഉଞॲཧ • ϑΝΠϧΛόοϑΝʔͱͯ࣋ͭ͠ • ಡΈࠐΈத͸ॻ͖ࠐ·ͳ͍ (ಡΈࠐΈ͕ऴΘΔ·Ͱ଴ͭ) • ॻ͖ࠐΈத͸ಡΈࠐ·ͳ͍ (ॻ͖ࠐΈ͕ऴΘΔ·Ͱ଴ͭ) • ϩοΫ(ͦͷ··)͸࢖͑ͳ͍ (σουϩοΫ͢Δ) actor FileCache { func write(_ data: Data) async func read() async -> Data? }
25. όοϑΝ (ϑΝΠϧ) εϨουB εϨουA ॻ͖ࠐΈ ಡΈࠐΈ
26. actor FileCache { let queue = DispatchQueue(label: "FileCacheQueue", qos: .utility) let semaphore = DispatchSemaphore(value: 1) func write(_ data: Data) async { await withCheckedContinuation { continuation in queue.async { self.semaphore.wait() Task { await self.write_(data) continuation.resume() self.semaphore.signal() } } } } func read() async -> Data? { await withCheckedContinuation { continuation in queue.async { self.semaphore.wait() Task { let data = await self.read_() continuation.resume(returning: data) self.semaphore.signal() } } } } • ඇಉظؔ਺͸ Serial Queue & Semaphore(or Mutex) • Actor͚ͩͰ͸ಉ࣌ಡΈॻ͖Λ๷͛ͳ͍৔߹͕͋Δ • อޢ͍ͨ͠Օॴ͕ඇಉظؔ਺
27. εϨουB εϨουA ϑΝΠϧ ೚ҙεϨου ࣌ؒ ॻ͖ࠐΈத Semaphore 1 0 wait() wait() DispatchQueue signal() Task ೚ҙεϨου 1 ಡΈࠐΈ 0 signal() write() read() 1 Queue Block…
28. ࢀߟจݙ • Core Audio Overview (Apple) https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/ CoreAudioOverview/WhatisCoreAudio/WhatisCoreAudio.html • Xcode 16 & Swift 6 ΩϟονΞοϓ: Swift Concurrencyͷجૅͱ࠷ॏཁϙΠϯτΛ૯෮श (QonceptInc) https://www.youtube.com/watch?v=jJgEtjx8KHY • ʲSwiftʳGrand Central Dispatch (GCD)ͱOperationQueue ·ͱΊ (@shiz) https://qiita.com/shiz/items/693241f41344a9df6d6f • Linuxͷsemaphoreͱmutexͷ࣮૷ (@watatuki) https://www.docswell.com/s/watatuki/Z7VNJG-2023-08-24-014308