Share
Embed
Transcript
43 slides
-
Low-latency sound in iOS Cyril Lashkevich iOS Solution Architect Viber
Media Inc. Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic Get ASBD to inspect audio format being received from H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic BUS 0 to speaker Get ASBD to inspect audio format being sent to H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic BUS 0 to speaker BUS 0 from app Set ASBD to indicate what you’re providing for play-out; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app Set ASBD to indicate what format you want your units to receive; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback Cyril Lashkevich Low-latency sound in iOS -
VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit
BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback InputCallback Cyril Lashkevich Low-latency sound in iOS -
AudioStreamBasicDescription Code struct AudioStreamBasicDescription { Float64 mSampleRate; .... UInt32 mChannelsPerFrame;
.... }; Cyril Lashkevich Low-latency sound in iOS -
Callbacks OSStatus AURenderCallback( void * inRefCon, AudioUnitRenderActionFlags * ioActionFlags, const
AudioTimeStamp * inTimeStamp, Int32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData); Cyril Lashkevich Low-latency sound in iOS -
Callbacks Used for both, input and output; Input callback calls
AudioUnitRender which provides recorded data from microphone; Render callback fills thr ioData bu↵ers with inNumberFrames of data; inTimeStamp contains timstamp for provided/requested frames. Cyril Lashkevich Low-latency sound in iOS -
AudioTimeStamp struct AudioTimeStamp { // The absolute sample frame time
Float64 mSampleTime; // mach_absolute_time UInt64 mHostTime; .... // Which fields are valid UInt32 mFlags; }; Cyril Lashkevich Low-latency sound in iOS -
Hardware bu↵er duration kAudioSessionProperty * Set: PreferredHardwareIOBufferDuration Get: CurrentHardwareIOBufferDuration; Minimal
latency we have from the HW bu↵erization. Cyril Lashkevich Low-latency sound in iOS -
Hardware bu↵er duration Default route, sample rate 44100 THW 2
{5.805 ms , 11.61 ms , 23.22 ms } Cyril Lashkevich Low-latency sound in iOS -
Hardware bu↵er duration Default route, sample rate 44100 THW 2
{5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Cyril Lashkevich Low-latency sound in iOS -
Hardware bu↵er duration Default route, sample rate 44100 THW 2
{5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Bu↵er duration in sec THW = N 256 mSampleRateHW ; N 2 {1, 2, 3, . . .} Cyril Lashkevich Low-latency sound in iOS -
Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024
Cyril Lashkevich Low-latency sound in iOS -
Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024
mSampleRateAPP = 16000 Cyril Lashkevich Low-latency sound in iOS -
Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024
mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms Cyril Lashkevich Low-latency sound in iOS -
Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024
mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms bu↵SizeAPP = 1024 · 16000 44100 = 371.52 bytes Cyril Lashkevich Low-latency sound in iOS -
Frames loss, route changes Frames loss Wrong thread syncronisation/design; High
CPU load; Up to 50ms missed frames. Cyril Lashkevich Low-latency sound in iOS -
Frames loss, route changes Frames loss Wrong thread syncronisation/design; High
CPU load; Up to 50ms missed frames. inTimeStamp->mSampleTime gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS -
Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms
missed samples; Bu↵er duration is unchanged; Cyril Lashkevich Low-latency sound in iOS -
Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms
missed samples; Bu↵er duration is unchanged; inTimeStamp->mSampleTime still gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS -
Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples;
Bu↵er duration can change; Cyril Lashkevich Low-latency sound in iOS -
Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples;
Bu↵er duration can change; inTimeStamp->mSampleTime is useless. inTimeStamp->mHostTime should be used for aproximate missed samples calculation. Cyril Lashkevich Low-latency sound in iOS -
Conclusion Process on HW sample rate if possible. Cyril Lashkevich
Low-latency sound in iOS -
Conclusion Process on HW sample rate if possible. Circular bu↵ers
are your friends. Cyril Lashkevich Low-latency sound in iOS -
Conclusion Process on HW sample rate if possible. Circular bu↵ers
are your friends. Lock-free rocks! Cyril Lashkevich Low-latency sound in iOS -
Conclusion Process on HW sample rate if possible. Circular bu↵ers
are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Cyril Lashkevich Low-latency sound in iOS -
Conclusion Process on HW sample rate if possible. Circular bu↵ers
are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Be careful with routes switch. Cyril Lashkevich Low-latency sound in iOS