音声配信アプリにおけるiOSを使った音声配信の全てと裏側

Transcript

1. Ի੠഑৴ΞϓϦʹ͓͚ΔiOSΛ࢖ͬ ͨԻ੠഑৴ͷશͯͱཪଆ iOSDC 2022 Track C 09/11 13:35ʙ
2. ࣗݾ঺հ Name: ԕ౻୓໻(͑Μͨ͘) Job: iOS / AndroidΤϯδχΞͳͲ Career: SIer໿6೥ 2018/3~ εϙʔπϚονϯάΞϓϦ 2019/3~ CBcloud ෺ྲྀITαʔϏε 2೥൒ 2021/12~ Voicy Twitter: @entaku_0818
3. Ի੠഑৴ϓϥοτϑΥʔϜ - Voicy https://corp.voicy.jp/ https://voicy.jp/ • Ի੠ίϯςϯπΛiOS/Android/WebͰఏڙ
4. VoicyͷΞϓϦ͸࠶ੜͱऩ࿥ͷ2छྨͷΞϓ ϦΛఏڙͯ͠·͢ ΠϯετʔϧϦϯΫ
5. ऩ࿥ͱ࠶ੜͰදཪҰମͰ͞·͟·ͳVoicyͷػೳ Ի੠࠶ੜ
   Ի੠ऩ࿥ ੜ์ૹ
   ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ
6. Ի੠αʔϏε্Ͱ͸ಛʹॏཁͳೋͭͷػೳ
   ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ Ի੠࠶ੜ
   Ի੠ऩ࿥ ੜ์ૹ
7. ར༻͍ͯ͠Δٕज़ Ի੠࠶ੜԻ੠ऩ࿥ ੜ์ૹ AVAudio Agora
8. AVAudioͱ͸ʁ
9. AVFAudioͱ͸ʁ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP
   &OHJOF • AVFoundation͔ΒAudioͷػೳΛ੾Γग़ͨ͠΋ͷͰ͢ • iOS14.5͔Β
10. AVFAudioͱ͸ʁ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP
    &OHJOF • VoicyͰར༻͍ͯ͠Δػೳ
11. Ի੠ͷऩ࿥
12. ࿥ԻͷྲྀΕ ։࢝ ࿥Իத ऴྃ - 1tapͰ࿥ԻՄೳͳઃܭ Ի੠ͷ೾ܗ৘ใදࣔ
13. ࿥Իݖݶ AVAudioSession.sharedInstance().requestRecordPermission { granted in self.handlePermissionGranted() } • ࿥ԻݖݶΛऔಘ https://developer.apple.com/documentation/avfaudio/avaudiosession/1616463- recordpermission
14. ηογϣϯઃఆ try audioSession.setCategory(.playAndRecord)ɹ try audioSession.setPreferredSampleRate(44100) try audioSession.setPreferredIOBufferDuration(defaultPreferredIOBufferDurat ion) try audioSession.setActive(true) https://developer.apple.com/documentation/avfaudio/avaudiosession • ࠶ੜऩ࿥ՄೳͳηογϣϯͰ։࢝ /P
    *NBHF
15. ࿥Ի։࢝ var componentDescription = AudioComponentDescription( componentType: OSType(kAudioUnitType_Output), componentSubType: OSType(kAudioUnitSubType_RemoteIO), componentManufacturer: OSType(kAudioUnitManufacturer_Apple), componentFlags: UInt32(0), componentFlagsMask: UInt32(0) ) let component = AudioComponentFindNext(nil, &componentDescription) var tempAudioUnit: AudioUnit? AudioComponentInstanceNew(component!, &tempAudioUnit) https://developer.apple.com/documentation/audiotoolbox • audioUnitΛੜ੒
16. ࿥Ի։࢝ var inputCallbackStruct = AURenderCallbackStruct( inputProc: recordingCallback, inputProcRefCon: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()) ) https://developer.apple.com/documentation/audiotoolbox • Callbackͷఆٛ
17. ࿥Ի։࢝ // ࿥ԻͷCallback private let recordingCallback: AURenderCallback = { (inRefCon, ioActionFlags, inTimeStamp, inBusNumber, frameCount, _) -> OSStatus in audioObject.handlerBufferData(bufferList: &bufferList, frameCount: frameCount) } • όοϑΝʔ৘ใΛऔಘ
18. ೾ܗͷऔಘ let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: UInt32(bufferSize)) buffers.append(buffer) waveFormHeights.append(buffer.waveFormHeight) • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
19. ೾ܗͷऔಘ 6*$PMMFDUJPO7JFX let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: UInt32(bufferSize)) buffers.append(buffer) waveFormHeights.append(buffer.waveFormHeight) • औಘͨ͠όοϑΝʔ৘ใΛ΋ͱʹԻྔΛϨϯμϦϯά
20. ࠶ฤू • ऩ࿥ͨ͠Ի੠σʔλΛ্ॻ͖
21. Ի੠ऩ࿥ͷऴྃ let settings: [String: Any] = [ AVSampleRateKey: Constants.Recording.sampleRate, AVFormatIDKey: kAudioFormatMPEG4AAC, // MP4ܗࣜͰอଘ AVNumberOfChannelsKey: Constants.Recording.numberOfChannel, AVEncoderAudioQualityKey: AVAudioQuality.high ] let audioFile = try AVAudioFile(forWriting: fileURL, settings: settings) • Local΁อଘ͠Ξοϓϩʔυ
22. AVAudioRecorder͸ར༻͍ͯ͠ͳ͍ https://developer.apple.com/documentation/avfaudio "7"VEJP4FTTJPO "7"VEJP3PVUJOH"SCJUFS "7"VEJP1MBZFS "7.*%*1MBZFS "7"VEJP3FDPSEFS "VEJP
    &OHJOF • ։ൃ౰ॳ͸audioUnitͰ༷ʑͳ͜ͱΛ΍Ζ͏ͱ͍ͯͨ͠ͷ͔΋͠Εͳ͍ • ͓ͦΒ͘೾ܗʹԻྔͷߴ͞Λऔಘ͢ΔͨΊʹར༻ BVEJP6OJU
23. Ի੠ͷฤू
24. ࠶ฤू • αʔϏεϙϦγʔ্Ͱ͖ͳ͍Α͏ʹ͍ͯ͠·͢ʂ
25. Ի੠࠶ੜ
26. Ի੠ͷ։࢝ Ի੠࠶ੜ։࢝ Ի੠࠶ੜத
27. Ի੠ͷ։࢝ func playPlayingChapter() { voicePlayer.play(url: fileURL, startTime: playingChapter.lastPlaytime.msecToSec(), speed: playSpeed.rawValue) } • νϟϓλʔ୯ҐͰͷ։࢝
28. Ի੠ͷఀࢭ func stopPlay() { player?.stop() player = nil } • Ի੠ͷఀࢭ
29. background࠶ੜ try audioSession.setCategory(.playback, mode: .default, options: []) https://developer.apple.com/documentation/avfaudio/avaudiosession/category
30. ੜ์ૹ
31. Agoraͱ͸ʁ
32. Agoraͱ͸ʁ https://www.agora.io/en/ https://jp.vcube.com/sdk
33. AgoraͰ͸ϏσΦ௨࿩΍Ի੠഑৴ͷػೳΛఏڙ https://www.agora.io/en/
34. ͦͷதͰ΋Live Steaming ͱ Real-time MessagingΛར༻ https://www.agora.io/en/
35. Voicyͷੜ์ૹͱ͸ʁ Ի੠ͷ࿈ܞ 
 Live Streaming (RTC) ͓ศΓ΍λΠτϧͷ࿈ܞ RealTimeMessage (RTM)
36. ੜ์ૹ
37. ੜ์ૹͷྲྀΕ ੜ์ૹ։࢝ ࿥Ի ऴྃ
38. ੜ์ૹ // RTC agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: EnvironmentConfig.agoraAppId, delegate: self) agoraKit?.enableAudioVolumeIndication(200, smooth: 3, report_vad: true) agoraKit?.setChannelProfile(AgoraChannelProfile.liveBroadcasting) // RTM agoraRtmKit = AgoraRtmKit(appId: EnvironmentConfig.agoraAppId, delegate: self) agoraRtmChannel = agoraRtmKit?.createChannel(withId: liveId, delegate: self) • Agora্ͰRTCͱRTMͰͦΕͧΕνϟϯωϧΛ࡞੒
39. ੜ์ૹ࿥Ի // ࿥Ի͸αʔόʔαΠυͰ࣮ࢪ͍ͯ͠·͢ // ͲΜͳॲཧ͔Θ͔ΔΑ͏ʹcurlͷAPIΛهࡌ͠·ͨ͠ curl --location --request GET 'https://api.agora.io/v1/apps/<appid>/ cloud_recording/resourceid/<resourceid>/sid/<sid>/mode/<mode>/query' \ • Agoraͷcloud_recording(࿥Ի)ͷػೳ
40. ͓ศΓϝοηʔδͷऔಘ func rtmKit(_ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage, fromPeer peerId: String) { messageReceived(message: message, uid: peerId) } ~~~ if let message = RTMMessage(rawValue: message.text), let uid = Int(uid) { switch message { case .hands_up: delegate?.onRaiseHand(userId: uid) case .hands_down: delegate?.onDroppedHand(userId: uid) case .invite: delegate?.onGuestInvited() case .canceled_invite: delegate?.onCancelGuestInvitation() case .reject: delegate?.onInvitaionRejected(userId: uid) case .set_audience: changeListener() case .mute: delegate?.onMute(uid: uid, muted: true) case .un_mute: delegate?.onMute(uid: uid, muted: false) case .stamp: delegate?.onStampReceived() case .edit_live_info: delegate?.onEditLiveInfo() case .letter: delegate?.onReceivedLetter(userId: uid) case .return_guest_to_listener: delegate?.onReturnGuestToListener() case .start: delegate?.onLiveStarted() } } • ϝοηʔδΛऔಘ༷͠ʑͳϦΞϧλΠϜॲཧΛ࣮ݱ
41. ੜ์ૹ
42. ੜ์ૹ • AgoraͰੜ์ૹʹࢀՃ self.agoraRtmChannel?.join(completion: { code in self.agoraKit?.setClientRole(agoraClientRole) self.agoraKit?.joinChannel(byToken: entity.agoraToken.rtcToken, channelId: String(liveID), info: nil, uid: UInt(entity.uid))
43. ੜ์ૹ • ͓ศΓΛૹΔ agoraRtmKit?.send(AgoraRtmMessage(text: text.rawValue), toPeer: uid)
44. Ի੠αʔϏε্(Voicy)Ͱ͸ಛʹॏཁͳೋͭͷػೳʹ ͍ͭͯ঺հ͠·ͨ͠
    ίϝϯτ ϋογϡλάݕ ࡧ ച্੥ٻ ࠩ͠ೖΕ ϓϨϛΞϜ์ૹ Ի੠࠶ੜ
    Ի੠ऩ࿥ ੜ์ૹ
45. VoicyͷԻ੠ॲཧͷࠓޙͷల๬
46. ࠓޙͷల๬ɹԻ੠࠶ੜ/Ի੠ऩ࿥ • Ի੠ऩ࿥ͷ඼࣭޲্ • ϚΠΫ - bluetoothϚΠΫͳͲ͸ి೾͕ׯবͯ͠͠ ·͍Ի੠͕௿͔ͬͨΓ͢Δ • ݱࡏ͸ϚΠΫΛ઀ଓ͠ͳ͍ঢ়ଶͰͷऩ࿥Λਪ঑ • ࿈ଓ࠶ੜ / ΦϑϥΠϯ࠶ੜͳͲ • Ϧεφʔ΁͞ΒʹศརͳԻ੠ମݧΛ
47. ࠓޙͷల๬ɹੜ์ૹ • ࿥Ի඼࣭޲্ • AgoraͷCloud-RecordingͩͱͲ͏͠ ͯ΋඼࣭͕ྑ͘ͳ͍ • େਓ਺Ͱ΋଱͑ΒΕΔੜ์ૹ΁ • AgoraͱαʔόʔαΠυؚΊͨΩϟύ γςΟͷ޲্
48. ͓·͚
49. ࠓ৽͘͠Ի੠ΞϓϦΛ࡞Δͱͨ͠ΒͲ͏͢ΔΜͩΖ͏ʁ 🤔 ΍ͬͺSwiftUI͔ʁ Swift Concurrencyͱ͔৽ ػೳ΋࢖͍͍ͨͳ
50. ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ https://github.com/entaku0818/VoiceMemo
51. αϯϓϧΛࢀߟʹͯ͠࡞Γ·ͨ͠ https://github.com/pointfreeco/swift-composable-architecture/tree/main/Examples/VoiceMemos ͱ͍͏Θ͚Ͱ࡞Γ·ͨ͠ʂ
52. ಛʹ঺հ͍ͨ͠ͱ͜Ζ
53. Ի੠ॲཧͷDeleagteॲཧͷ؅ཧͷ೉͠͞ • େ͖ͳControllerͰશͯΛίϯτϩʔϧ͕ͪ͠ • ֤ը໘ͰDelegateͰॲཧΛॻ͖͕ͪ
54. Ի੠ͳͲͰ͸ಛʹଟ͍DelegateॲཧΛ·ͱΊ ͯΔ https://github.com/entaku0818/VoiceMemo private final class Delegate: NSObject, AVAudioPlayerDelegate, Sendable { let didFinishPlaying: @Sendable (Bool) -> Void let decodeErrorDidOccur: @Sendable (Error?) -> Void let player: AVAudioPlayer init( url: URL, didFinishPlaying: @escaping @Sendable (Bool) -> Void, decodeErrorDidOccur: @escaping @Sendable (Error?) -> Void ) throws { self.didFinishPlaying = didFinishPlaying self.decodeErrorDidOccur = decodeErrorDidOccur self.player = try AVAudioPlayer(contentsOf: url) super.init() self.player.delegate = self } func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) { self.didFinishPlaying(flag) } func audioPlayerDecodeErrorDidOccur(_ player: AVAudioPlayer, error: Error?) { self.decodeErrorDidOccur(error) } }
55. liveͱ͍͏ঢ়ଶͰDelegateΛఆٛ Ի੠։࢝࣌ͷॲཧΛެ։ https://github.com/entaku0818/VoiceMemo struct AudioPlayerClient { var play: @Sendable (URL) async throws -> Bool } extension AudioPlayerClient { static let live = Self { url in let stream = AsyncThrowingStream<Bool, Error> { continuation in do { let delegate = try Delegate( url: url, didFinishPlaying: { successful in continuation.yield(successful) continuation.finish() }, decodeErrorDidOccur: { error in continuation.finish(throwing: error) } ) delegate.player.play() continuation.onTermination = { _ in delegate.player.stop() } } catch { continuation.finish(throwing: error) } } return try await stream.first(where: { _ in true }) ?? false } } 4XJGU
    $PODVSSFODZ Ͱ࣮૷
56. ReducerͰը໘ͰඞཁͳॲཧΛఆٛ https://github.com/entaku0818/VoiceMemo switch state.mode { case .notPlaying: state.mode = .playing(progress: 0) return .run { [url = state.url] send in let start = environment.mainRunLoop.now async let playAudio: Void = send( .audioPlayerClient(TaskResult { try await environment.audioPlayer.play(url) }) ) for try await tick in environment.mainRunLoop.timer(interval: 0.5) { await send(.timerUpdated(tick.date.timeIntervalSince(start.date))) } } .cancellable(id: PlayID.self, cancelInFlight: true) case .playing: state.mode = .notPlaying return .cancel(id: PlayID.self) }
57. ࢀߟࢿྉ
58. ࢀߟࢿྉ • agoraΛ࢖ͬͯϥΠϒ഑৴ػೳΛ1ϲ݄൒ͰϦϦʔεͨ͠࿩ • εϥΠυ • ࠓ೔͔Β෼͔Δ AVAudioEngineͷશͯ • εϥΠυ • ࣮૷ͨ͘͠ͳΔԻ੠ฤू • εϥΠυ • աڈͷԻ੠ʹؔΘΔiOSDCͷηογϣϯ
59. ࢀߟࢿྉ • ΍ͬͺΓ࠷ޙ͸ެࣜ • https://developer.apple.com/documentation/avfaudio • https://docs.agora.io/en
60. એ఻ʂ
61. https://voicy.jp/channel/1305 VoicyͷΤϯδχΞ͕ӡӦ͍ͯ͠Δνϟϯωϧ voi-chordௌ͍͍ͯͩ͘͞ʂ
62. iOSΤϯδχΞΛืू͍ͯ͠·͢ • Ի੠޷͖ͳΤϯδχΞืूதͰ͢ʂڵຯ͋Δํ͸DM͍ͩ͘͞ʂ Notion Twitter: @entaku_0818