Share
Embed
Swiftで高速フーリエ変換してオーディオビジュアライザーを作る / iOSDC Japan 2024 Day1 Track D
Kyome (Takuto Nakamura)
August 24, 2024
Transcript
43 slides
-
Kyome iOSDC Japan 2024 - day1 Track D 2024/08/23 SwiftͰߴϑʔϦΤมͯ͠
ΦʔσΟΦϏδϡΞϥΠβʔΛ࡞Δ 1 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࣗݾհ
IUUQTLZPNFJP ,ZPNF αΠϘζͰ kintoneϞόΠϧ ͷiOS։ൃΛ୲ macOS͚ͷϢʔςΟϦςΟΞϓϦ։ൃऀ ੵۃతʹOSSΛ։ൃӡ༻ 2 Kyome -
iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͱ
ԻָԻͷϦζϜपಛੑΛࢹ֮తʹදݱͨ͠ͷ Dynamic IslandGarageBandɺϘΠεϝϞͳͲͰ͓ೃછΈ 4 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͱ
ԻָԻͷϦζϜपಛੑΛࢹ֮తʹදݱͨ͠ͷ Dynamic IslandGarageBandɺϘΠεϝϞͳͲͰ͓ೃછΈ 5 ͔͍͍ͬ͜ʂ࡞ͬͯΈ͍ͨʂ -
iOSDC Japan 2024 - day1 Track D #iosdc #d ΦʔσΟΦϏδϡΞϥΠβʔͷ࣮ํ
1. AVAudioEngineͰԻָΛ࠶ੜ͠ɺԻݯσʔλΛऔಘ͢Δ 2. AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ 3. ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ 7 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ
1/3 1. AVAudioEngineͰԻָΛ࠶ੜ͢Δ AVFoundationͷAVAudioEngineͱAVAudioPlayerNodeΛ༻͍Δ attach, connect, scheduleFile, start, playͷ࣮ߦॱং݅ʹҙ import AVFoundation let engine = AVAudioEngine() let playerNode = AVAudioPlayerNode() func play() throws { let url = Bundle.main.url(forResource: "sound", withExtension: "mp3")! let file = try AVAudioFile(forReading: url) let sampleRate = Float(file.processingFormat.sampleRate) engine.attach(playerNode) engine.connect(playerNode, to: engine.mainMixerNode, format: file.processingFormat) playerNode.scheduleFile(file, at: nil) try engine.start() playerNode.play() } 9 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ
2/3 2. Իָͷ࠶ੜʹ߹ΘͤͯԻݯσʔλΛऔಘ͢Δ ࠶ੜલʹAVAudioPlayerNodeʹinstallTapͰbu ff erSizeΛಡΈࠐΉͨͼʹ ࣮ߦ͞ΕΔॲཧΛొ͓ͯ͘͠ ҰൠతͳԻָϑΝΠϧαϯϓϧϨʔτ͕ 44.1kHz ͳͷͰɺ bu ff erSizeΛ 2048 ʹ͢Δͱ̍ඵؒʹ 22 ճඳըͷߋ৽Λ͢Δ͜ͱʹͳΔ let file = try AVAudioFile(forReading: url) let sampleRate = Float(file.processingFormat.sampleRate) engine.attach(playerNode) engine.connect(playerNode, to: engine.mainMixerNode, format: file.processingFormat) playerNode.installTap(onBus: .zero, bufferSize: 2048, format: nil) { buffer, _ in if let data = buffer.floatChannelData { // ԻݯσʔλΛ༻͍ͨҙͷॲཧ } } 10 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AVAudioEngineͰԻָΛ࠶ੜ͠ԻݯσʔλΛऔಘ͢Δ
3/3 3. ԻָΛఀࢭ͢Δ stop, removeTap, disconnectNodeOutput, detachͷ࣮ߦॱং݅ʹҙ ಛʹinstallTap͍ͯ͠ͳ͍ͷʹremoveTapͨ͠Γattach͍ͯ͠ͳ͍ͷʹdetach ͠Α͏ͱ͢ΔͱΫϥογϡ͢Δ͕ɺisPlayingisRunningͷΑ͏ͳϑϥάͳ͍ ͷͰࣗݾཧ͢Δඞཁ͕͋Δ func stop() { if playerNode.isPlaying { playerNode.stop() } playerNode.removeTap(onBus: .zero) if engine.isRunning { engine.stop() } engine.disconnectNodeOutput(playerNode) engine.detach(playerNode) } 11 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰ
ৼ෯εϖΫτϧΛࢉग़͢Δ 12 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
1/7 ϑʔϦΤڃల։ɿ ͋ΒΏΔपظతͳ؆୯ͳͷॏͶ߹ΘͤͰද͢͜ͱ͕Ͱ͖Δ ʢ؆୯ͳʹجຊपͷഒͷपͷਖ਼ݭ༨ݭʣ ʹ ʴ ʴ 15 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
2/7 ϑʔϦΤมɿ पظతͰͳ͍Ͱ͋ͬͯɺಛఆͷ۠ؒʹ͠पظతͰ͋ΔͱΈͳͯ͠ ༷ʑͳपͷਖ਼ݭɾ༨ݭʹղͯ͠ղੳ͕Մೳ Իڹಛੑͷղੳɿ Իͷೖྗ৴߸Λҙͷ۠ؒͰ۠ΓϑʔϦΤมΛߦ͏͜ͱͰɺ ֤۠ؒʹ͓͚ΔपಛੑΛղੳՄೳ ૭ɿ ϑʔϦΤมΛߦ͏۠ؒͷ͜ͱ 16 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
3/7 ϑʔϦΤมͷղೳɿ ϑʔϦΤมʹΑΓແݶճͷղΛߦ͑ݩͷͱҰக͢Δ͕ ༗ݶճͷ߹ۙࣅͱͳΔ 㲈 ʴ ʴ ʹ 17 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
4/7 ૭ؔɿ ૭ͷͷෆ࿈ଓੑʹىҼ͢ΔϊΠζΛআڈ͢ΔͨΊʹ ۠ؒͷΛ0ʹ͢ΔΑ͏ͳ૭ؔΛ͔͚Δ 18 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
5/7 ૭ؔͷछྨɿ पղೳͱμΠφϛοΫϨϯδʢॲཧՄೳͳ৴߸ͷ࠷େͱ࠷খͷ ൺʣͷτϨʔυΦϑͷதͰ༷ʑͳ૭͕ؔߟҊ͞Ε͍ͯΔ 19 Ψε૭ ϋϯ૭ ϋϛϯά૭ ϒϥοΫϚϯ૭ ͥΜͿಉ͡͡Όͳ͍Ͱ͔͢ -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
6/7 ৼ෯εϖΫτϧɿ ϑʔϦΤมͨ݁͠Ռ͔Βɺԣ࣠Λपɺॎ࣠Λͦͷपͷڧ ͱͨ͠άϥϑ͕ϓϩοτͰ͖Δ 20 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ߴϑʔϦΤมͱ
7/7 ࢄϑʔϦΤมɿ ίϯϐϡʔλʔͰ࿈ଓؔʢແݶͷղೳʣѻ͑ͳ͍ͷͰࢄԽͯ͠ ༗ݶݸͷྻσʔλʹͯ͠ϑʔϦΤมΛߦ͏ ߴϑʔϦΤมɿ ࢄϑʔϦΤมେมԋࢉʹ͕͔͔࣌ؒΔͨΊɺ ΞϧΰϦζϜΛͯ͠ԋࢉΛߴԽͨ͠ͷ͕ߴϑʔϦΤม FFTɿFast Fourier Transformͷུশ ߴϑʔϦΤมશʹཧղͨ͠ 21 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰ
ৼ෯εϖΫτϧΛࢉग़͢Δ 22 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
0/7 Accelerate Frameworkɿ CPUͷϕΫτϧॲཧػೳΛ׆༻ͯ͠ɺߴੑೳ͔ͭߴޮͳԋࢉΛ࣮ߦͰ͖Δ vDSP (digital signal processing on vectors)ɿ σδλϧ৴߸ॲཧ͚ʹྻͰͷ൚༻ԋࢉʹ࠷దԽ͞ΕͨAPIΛఏڙ͍ͯ͠Δ ߹ܭɺฏۉɺ࠷େͳͲͷੵؔΛ͡Ίͱ͠ɺϑʔϦΤม 2࣍ϑΟϧλͷΑ͏ͳॲཧΧόʔ 23 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
1/7 1. vDSPͰϑʔϦΤมͷ४උΛ͢Δ class FFT { let fftFullSize: vDSP_Length let fftHalfSize: vDSP_Length let mLog2N: vDSP_Length var fftSetup: FFTSetup? init(size: Int) { // ϑʔϦΤมΛ͔͚Δ૭ͷαΠζΛఆٛ͢Δ fftFullSize = vDSP_Length(size) fftHalfSize = vDSP_Length(size / 2) // ૭ͷαΠζ͕̎ͷԿ͔Λࢉग़͢Δ mLog2N = vDSP_Length(log2(Double(size)).rounded() + 1.0) // FFTSetupΛॳظԽ͢Δ fftSetup = vDSP_create_fftsetup(mLog2N, FFTRadix(kFFTRadix2)) } 24 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
2/7 2. FFTSetupͷഁغΛఆ͓ٛͯ͘͠ class FFT { // ... deinit { // OpaquePointerΛ͍ͬͯΔͷͰϝϞϦͷղ์ΛΕͣʹ vDSP_destroy_fftsetup(fftSetup) } 25 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
3/7 3. Իͷೖྗ৴߸ʹ૭ؔΛ͔͚Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ૭ؔ༻ͷྻσʔλΛ࡞Δ let windowData = UnsafeMutablePointer<Float>.allocate(capacity: Int(fftFullSize)) // UnsafeMutablePointerΛ͏ͷͰϝϞϦͷղ์ΛΕͣʹ defer { windowData.deallocate() } // ྻʹ૭ؔͷΛ٧ΊΔʢྫͱͯ͠ϋϯ૭Λ༻͍Δʣ vDSP_hann_window(windowData, fftFullSize, 0) // Իͷೖྗ৴߸ʹ૭ؔΛ͔͚Δʢ֤ཁૉʹ૭ؔͷΛֻ͚߹ΘͤΔʣ vDSP_vmul(audioData, 1, windowData, 1, windowData, 1, fftFullSize) 26 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
4/7 4. ૭ؔΛ͔͚ͨԻͷೖྗ৴߸Λෳૉͷྻʹม͢Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // θϩͰຒΊΒΕͨྻΛ༻ҙ͢Δ let zeroData = UnsafeMutablePointer<Float>.allocate(capacity: Int(fftFullSize)) defer { zeroData.deallocate() } vDSP_vclr(zeroData, 1, fftFullSize) // ࣮෦͕૭ؔΛ͔͚ͨԻͷೖྗ৴߸ɺڏ෦͕θϩຒΊྻͷෳૉྻΛ࡞Δ var dspSplitComplex = DSPSplitComplex( realp: windowData, imagp: zeroData ) 27 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
5/7 5. ࡞ͬͨෳૉͷྻʹߴϑʔϦΤมΛ͔͚Δ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // ߴϑʔϦΤม vDSP_fft_zrip(fftSetup, &dspSplitComplex, 1, mLog2N, FFTDirection(FFT_FORWARD)) 28 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
6/7 6. ߴϑʔϦΤมͷ݁Ռ͔Βৼ෯εϖΫτϧΛࢉग़͢Δʢ1/2ʣ ͜͜Ͱɺෳૉʹର͢Δઈର Ͱ͋Δ ͨͩɺ͜ΕͰಘΒΕΔͷਖ਼ෛ྆ଆͷৼ෯εϖΫτϧͰ͋Γ ৼ෯ͳͨΊɺಘΒΕͨৼ෯Λ̎ഒ͠ยଆ͚ͩΛ༻͢Δ ৼ෯ = | ϑʔϦΤมͷ݁Ռʢෳૉʣ ཁૉʢ૭ͷαΠζʣ | (࣮෦)2 + (ڏ෦)2 29 🤔 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
6/7 6. ߴϑʔϦΤมͷ݁Ռ͔Βৼ෯εϖΫτϧΛࢉग़͢Δʢ2/2ʣ class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // ϑʔϦΤมͷ݁ՌΛཁૉͰׂΔʢ͜ͷ࣌Ͱ૭ͷαΠζͷ͚ͩΛ༻͢Δʣ var fftNormFactor = Float(fftFullSize) vDSP_vsdiv(dspSplitComplex.realp, 1, &fftNormFactor, dspSplitComplex.realp, 1, fftHalfSize) vDSP_vsdiv(dspSplitComplex.imagp, 1, &fftNormFactor, dspSplitComplex.imagp, 1, fftHalfSize) // ઈରΛࢉग़͢Δ => ৼ෯͕ಘΒΕΔ var magnitudeData = [Float](repeating: .zero, count: Int(fftHalfSize)) vDSP_zvabs(&dspSplitComplex, 1, &magnitudeData, 1, fftHalfSize) // ৼ෯Λ̎ഒ͢Δ var fftFactor = Float(2) vDSP_vsmul(magnitudeData, 1, &fftFactor, &magnitudeData, 1, fftHalfSize) 30 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AccelerateͷߴϑʔϦΤมͰৼ෯εϖΫτϧΛࢉग़͢Δ
7/7 7. ֤ৼ෯ʹରԠ͢ΔपΛࢉग़͢Δ प = αϯϓϧϨʔτ ཁૉʢ૭ͷαΠζʣ × ཁૉͷΠϯσοΫε class FFT { // ... func compute(sampleRate: Float, audioData: UnsafePointer<Float>) -> [(Float, Float)] { // ... // 1͔Β૭ͷαΠζͷ·ͰͷྻΛ࡞Δ var hertsData: [Float] = vDSP.ramp(withInitialValue: 1, increment: 1, count: Int(fftHalfSize)) // αϯϓϧϨʔτΛཁૉͰׂΔ var hertsFactor = sampleRate / Float(fftFullSize) // पͷྻΛ࡞Δ vDSP_vsmul(hertsData, 1, &hertsFactor, &hertsData, 1, fftHalfSize) return zip(hertsData, magnitudeData).map { ($0, $1) } } 31 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛ
GraphicsContextͰՄࢹԽ͢Δ 32 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ
1/5 var body: some View { Canvas { context, size in let path = Path(ellipseIn: CGRect(origin: .zero, size: size)) context.fill(path, with: .color(.blue)) context.stroke(path, with: .color(.red), lineWidth: 5) } } SwiftUIͷCanvasΛ༻͍ͯGraphicsContextΛૢ࡞͢Δ fi llstrokeΛ༻͍ͯPathΛඳ͍͍ͯ͘ 33 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ
2/5 Canvas { context, size in let path = Path(ellipseIn: CGRect(origin: .zero, size: size)) context.fill(path, with: .color(.blue)) context.stroke(path, with: .color(.red), lineWidth: 5) } Path { path in path.addEllipse(in: CGRect(x: 0, y: 0, width: 50, height: 50)) } .stroke(lineWidth: 5) .fill() .foregroundStyle(Color.red) CanvasͱPathͷҧ͍ Canvasදࣔ͢ΔྖҬαΠζ͕༩͑ΒΕΔ Canvasಉ͡Pathʹରͯ͠ fi llͱstrokeͰృΓΛม͑ΒΕΔ 34 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ
3/5 struct AudioVisualizedView: View { let values: [Float] var body: some View { Canvas { context, size in let unit = size.width / CGFloat(values.count) let width = 0.8 * unit values.indices.forEach { index in let x = unit * CGFloat(index) let height = size.height * CGFloat(values[index]) let y = 0.5 * (size.height - height) let path = Path(CGRect(x: x, y: y, width: width, height: height)) context.fill(path, with: .color(.primary)) } } } } ৼ෯ͷ̌ʙ̍ͷൣғͰऔಘͰ͖ΔͷͰɺ ৼ෯εϖΫτϧͷྻ͔ΒάϥϑΛඳը͢Δ 35 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ
4/5 ϑϦʔBGMʮSUMMER TRIANGLEʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 36 ࣮ࡍʹಈ͔ͯ͠ΈΔͱ͜Μͳײ͡ -
iOSDC Japan 2024 - day1 Track D #iosdc #d ࢉग़ͨ͠ৼ෯εϖΫτϧΛGraphicsContextͰՄࢹԽ͢Δ
5/5 औಘͨ͠ৼ෯εϖΫτϧΛͦͷ··දࣔ͢Δͱͭ·Βͳ͍ ৼ෯Λఆഒͨ͠Γɺಈ͖͕໘ന͍पଳ͚ͩʹߜͬͯඳըͨ͠Γ͢Δ ώτͷՄௌҬ 20~20000hz ͳͷͰͦͷൣғͰΑ͍ ܦݧଇతʹ 100~3000hz ͘Β͍Ͱे ϑϦʔBGMʮSUMMER TRIANGLEʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 37 -
iOSDC Japan 2024 - day1 Track D #iosdc #d ͳΜͱͳ͔͚ͬͨ͘Ͳ
ֶ࣮ϜζΧγΠ😓 38 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AudioVisualizerKit
ԻָΛ࠶ੜ͠ͳ͕ΒߴϑʔϦΤมΛͯ͠ΦʔσΟΦϏδϡΞϥΠβʔΛ දࣔ͢ΔͨΊͷΩοτΛϥΠϒϥϦʹ͠·ͨ͠ʂʂ https://github.com/Kyome22/AudioVisualizerKit ϑϦʔBGMʮ͠ΎΘ͠ΎΘϋχʔϨϞϯ350mlʯʗ࡞ʢฤʣۂ ɿ ͠ΌΖ͏ 41 -
iOSDC Japan 2024 - day1 Track D #iosdc #d AudioVisualizerKit
ԻָΛ࠶ੜͭͭ͠ৼ෯εϖΫτϧͱԻѹΛࢉग़ͯ͘͠ΕΔAudioAnalyzerͱ ࢉग़ͨ͠σʔλΛجʹՄࢹԽͯ͘͠ΕΔAmplitudeSpectrumViewΛఏڙ import AudioVisualizerKit import SwiftUI struct ContentView: View { let audioAnalyzer = AudioAnalyzer(fftSize: 2048, windowType: .hannWindow) var body: some View { AmplitudeSpectrumView( shapeType: .straight, magnitudes: audioAnalyzer.magnitudes, range: 0 ..< 128, rms: audioAnalyzer.rms ) .onAppear { let url = Bundle.main.url(forResource: "sound", withExtension: "mp3")! try? audioAnalyzer.prepare(url: url) try? audioAnalyzer.play() } .onDisappear { audioAnalyzer.stop() } } } 42 ͬͯΈͯͶ -
iOSDC Japan 2024 - day1 Track D #iosdc #d Q
& A 43 Thank you!