Share
Embed
音声配信アプリにおけるiOSを使った音声配信の全てと裏側
entaku
September 11, 2022
Transcript
62 slides
-
ࣗݾհ Name: ԕ౻(͑Μͨ͘) Job: iOS / AndroidΤϯδχΞͳͲ Career: SIer6 2018/3~
εϙʔπϚονϯάΞϓϦ 2019/3~ CBcloud ྲྀITαʔϏε 2 2021/12~ Voicy Twitter: @entaku_0818 -
ηογϣϯઃఆ try audioSession.setCategory(.playAndRecord)ɹ try audioSession.setPreferredSampleRate(44100) try audioSession.setPreferredIOBufferDuration(defaultPreferredIOBufferDurat ion) try audioSession.setActive(true)
https://developer.apple.com/documentation/avfaudio/avaudiosession • ࠶ੜऩՄೳͳηογϣϯͰ։࢝ /P*NBHF -
Ի։࢝ var componentDescription = AudioComponentDescription( componentType: OSType(kAudioUnitType_Output), componentSubType: OSType(kAudioUnitSubType_RemoteIO), componentManufacturer:
OSType(kAudioUnitManufacturer_Apple), componentFlags: UInt32(0), componentFlagsMask: UInt32(0) ) let component = AudioComponentFindNext(nil, &componentDescription) var tempAudioUnit: AudioUnit? AudioComponentInstanceNew(component!, &tempAudioUnit) https://developer.apple.com/documentation/audiotoolbox • audioUnitΛੜ -
Ի։࢝ var inputCallbackStruct = AURenderCallbackStruct( inputProc: recordingCallback, inputProcRefCon: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()) )
https://developer.apple.com/documentation/audiotoolbox • Callbackͷఆٛ -
Ի։࢝ // ԻͷCallback private let recordingCallback: AURenderCallback = { (inRefCon,
ioActionFlags, inTimeStamp, inBusNumber, frameCount, _) -> OSStatus in audioObject.handlerBufferData(bufferList: &bufferList, frameCount: frameCount) } • όοϑΝʔใΛऔಘ -
ܗͷऔಘ 6*$PMMFDUJPO7JFX let pcmBuffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: UInt32(bufferSize)) buffers.append(buffer)
waveFormHeights.append(buffer.waveFormHeight) • औಘͨ͠όοϑΝʔใΛͱʹԻྔΛϨϯμϦϯά -
Իऩͷऴྃ let settings: [String: Any] = [ AVSampleRateKey: Constants.Recording.sampleRate, AVFormatIDKey:
kAudioFormatMPEG4AAC, // MP4ܗࣜͰอଘ AVNumberOfChannelsKey: Constants.Recording.numberOfChannel, AVEncoderAudioQualityKey: AVAudioQuality.high ] let audioFile = try AVAudioFile(forWriting: fileURL, settings: settings) • Localอଘ͠Ξοϓϩʔυ -
ੜ์ૹ // RTC agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: EnvironmentConfig.agoraAppId, delegate: self) agoraKit?.enableAudioVolumeIndication(200,
smooth: 3, report_vad: true) agoraKit?.setChannelProfile(AgoraChannelProfile.liveBroadcasting) // RTM agoraRtmKit = AgoraRtmKit(appId: EnvironmentConfig.agoraAppId, delegate: self) agoraRtmChannel = agoraRtmKit?.createChannel(withId: liveId, delegate: self) • Agora্ͰRTCͱRTMͰͦΕͧΕνϟϯωϧΛ࡞ -
ੜ์ૹԻ // ԻαʔόʔαΠυͰ࣮ࢪ͍ͯ͠·͢ // ͲΜͳॲཧ͔Θ͔ΔΑ͏ʹcurlͷAPIΛهࡌ͠·ͨ͠ curl --location --request GET 'https://api.agora.io/v1/apps/<appid>/
cloud_recording/resourceid/<resourceid>/sid/<sid>/mode/<mode>/query' \ • Agoraͷcloud_recording(Ի)ͷػೳ -
͓ศΓϝοηʔδͷऔಘ func rtmKit(_ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage, fromPeer peerId:
String) { messageReceived(message: message, uid: peerId) } ~~~ if let message = RTMMessage(rawValue: message.text), let uid = Int(uid) { switch message { case .hands_up: delegate?.onRaiseHand(userId: uid) case .hands_down: delegate?.onDroppedHand(userId: uid) case .invite: delegate?.onGuestInvited() case .canceled_invite: delegate?.onCancelGuestInvitation() case .reject: delegate?.onInvitaionRejected(userId: uid) case .set_audience: changeListener() case .mute: delegate?.onMute(uid: uid, muted: true) case .un_mute: delegate?.onMute(uid: uid, muted: false) case .stamp: delegate?.onStampReceived() case .edit_live_info: delegate?.onEditLiveInfo() case .letter: delegate?.onReceivedLetter(userId: uid) case .return_guest_to_listener: delegate?.onReturnGuestToListener() case .start: delegate?.onLiveStarted() } } • ϝοηʔδΛऔಘ༷͠ʑͳϦΞϧλΠϜॲཧΛ࣮ݱ -
ੜ์ૹ • AgoraͰੜ์ૹʹࢀՃ self.agoraRtmChannel?.join(completion: { code in self.agoraKit?.setClientRole(agoraClientRole) self.agoraKit?.joinChannel(byToken: entity.agoraToken.rtcToken,
channelId: String(liveID), info: nil, uid: UInt(entity.uid)) -
ࠓޙͷలɹԻ࠶ੜ/Իऩ • Իऩͷ্࣭ • ϚΠΫ - bluetoothϚΠΫͳͲి͕ׯবͯ͠͠ ·͍Ի͕͔ͬͨΓ͢Δ • ݱࡏϚΠΫΛଓ͠ͳ͍ঢ়ଶͰͷऩΛਪ
• ࿈ଓ࠶ੜ / ΦϑϥΠϯ࠶ੜͳͲ • Ϧεφʔ͞ΒʹศརͳԻମݧΛ -
ԻͳͲͰಛʹଟ͍DelegateॲཧΛ·ͱΊ ͯΔ https://github.com/entaku0818/VoiceMemo private final class Delegate: NSObject, AVAudioPlayerDelegate, Sendable
{ let didFinishPlaying: @Sendable (Bool) -> Void let decodeErrorDidOccur: @Sendable (Error?) -> Void let player: AVAudioPlayer init( url: URL, didFinishPlaying: @escaping @Sendable (Bool) -> Void, decodeErrorDidOccur: @escaping @Sendable (Error?) -> Void ) throws { self.didFinishPlaying = didFinishPlaying self.decodeErrorDidOccur = decodeErrorDidOccur self.player = try AVAudioPlayer(contentsOf: url) super.init() self.player.delegate = self } func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) { self.didFinishPlaying(flag) } func audioPlayerDecodeErrorDidOccur(_ player: AVAudioPlayer, error: Error?) { self.decodeErrorDidOccur(error) } } -
liveͱ͍͏ঢ়ଶͰDelegateΛఆٛ Ի։࢝࣌ͷॲཧΛެ։ https://github.com/entaku0818/VoiceMemo struct AudioPlayerClient { var play: @Sendable (URL)
async throws -> Bool } extension AudioPlayerClient { static let live = Self { url in let stream = AsyncThrowingStream<Bool, Error> { continuation in do { let delegate = try Delegate( url: url, didFinishPlaying: { successful in continuation.yield(successful) continuation.finish() }, decodeErrorDidOccur: { error in continuation.finish(throwing: error) } ) delegate.player.play() continuation.onTermination = { _ in delegate.player.stop() } } catch { continuation.finish(throwing: error) } } return try await stream.first(where: { _ in true }) ?? false } } 4XJGU$PODVSSFODZ Ͱ࣮ -
ReducerͰը໘ͰඞཁͳॲཧΛఆٛ https://github.com/entaku0818/VoiceMemo switch state.mode { case .notPlaying: state.mode = .playing(progress:
0) return .run { [url = state.url] send in let start = environment.mainRunLoop.now async let playAudio: Void = send( .audioPlayerClient(TaskResult { try await environment.audioPlayer.play(url) }) ) for try await tick in environment.mainRunLoop.timer(interval: 0.5) { await send(.timerUpdated(tick.date.timeIntervalSince(start.date))) } } .cancellable(id: PlayID.self, cancelInFlight: true) case .playing: state.mode = .notPlaying return .cancel(id: PlayID.self) } -
ࢀߟࢿྉ • agoraΛͬͯϥΠϒ৴ػೳΛ1ϲ݄ͰϦϦʔεͨ͠ • εϥΠυ • ࠓ͔Β͔Δ AVAudioEngineͷશͯ • εϥΠυ
• ࣮ͨ͘͠ͳΔԻฤू • εϥΠυ • աڈͷԻʹؔΘΔiOSDCͷηογϣϯ