Share
Embed
Taking trends in music app development into the future mobile ecosystem
At a non-public conference
Atsushi Eno
April 09, 2025
Transcript
51 slides
-
Taking trends in music app development into the future mobile
ecosystem atsushieno -
Agenda Today I will be focusing on - audio plugin
system (effects and synths) for audio apps - on mobile platforms - taking best from open technology and FLOSS - along with audio developers' tech. trends (kinda) images on the slides mostly has links to the originals (website, product, etc.) -
Tech Trends in Audio "development" ? They do not mean
"majority" (more like "hot topics") • AI-based composition (Suno, Udio, Yue, DiffRhythm, etc.) • AI-assisted tooling (trackmaking / production tools / virtual singers) • spatial audio • WebView as GUI • CLAP plugin format • MIDI 2.0 • C++ I check those trends at: audio.dev, theaudioprogrammer.com etc. -
many different kinds of audio / music software music &
video player (files, streaming) -
many different kinds of audio / music software games (music,
sound effects, character voices. interactive audio) -
many different kinds of audio / music software video and
voice calls (or: karaoke, sessions) -
many different kinds of audio / music software audio recorder
& editor, streaming content authoring -
many different kinds of audio / music software creative tools:
DAW, DJ tools, audio plugins (instruments + effects) -
How LIVE is your audio effects? (C++ or not) Music
player: user may change effects interactively Games: apply effects as user controls DAW: apply effects as user plays Live streaming: change effects while user speaks They are all LIVE (some are even real-time, some are even low-latency) (also their slides) -
How LIVE is your audio app? (C++ or not) accept
glitches and delays? Use C++ (or C or Rust or Zig...) Use any language! (JS, Java, Python, Go...) YES GC JIT dynamic types interpreter is it live? NO YES NO -
How LIVE is your audio app? (C++ or not) accept
glitches and delays? Use any language! (JS, Java, Python, Go...) YES NO GC JIT dynamic types interpreter is it live? NO YES Use C++ (or C or Rust or Zig...) -
should we write everything in C++ (etc.) ? not really.
in audio I/O thread? Use C++ etc. Use any language! YES NO UI malloc locks I/O React/JS Flutter/Dart Unity/C# -
should we write everything in C++ (etc.) ? not really.
in audio I/O thread? Use C++ etc. Use any language! YES NO UI malloc locks I/O React/JS Flutter/Dart Unity/C# -
...do they use any other language? well, actually... it is
changing though (WebView) does it deal with audio I/O? Use C++ etc. Use C++ regardless 😛 YES NO Qt JUCE GUI OpenGL VSTGUI Trend -
Explaining some trends AI technology: audio developers care about realtime
safety • in RT-safe code: RTNeural etc. • in non-RT-safe code: OnnxRuntime etc. with RT thread C++: how we can achieve RT safety • RT-safe algorithms • memory operations (allocation, atomic shared pointers / hazard pointers) • related toolchains and stdlibs e.g. RealtimeSanitizer -
What are audio plugins? effects? synths? Audio Plugins = Effects
+ Synths (+ misc.) -
What are audio plugins? effects? synths? Effects Synths Control audio
outputs by parameters and - audio inputs (effects) - instrument inputs (MIDI keyboard etc.) Not very different -
What are audio plugins? effects? synths? What can be controlled?
What can be an effect? • gain (volume), pan, envelope • pitch shifter, tuner • reverb • delay echo • equalizer • distortion • phazer • vocoder • ... What are instruments? • samplers • synthesizers ◦ wavetable ◦ additive ◦ subtractive ◦ FM ◦ ... -
Two kinds of audio effect systems • Platform (OS) internal
effects • Open audio plugin formats / apps -
Platform (OS) Internal Effects • achieve basic effects: "bass boost",
"equalizer", "reverb", "denoise" ... • only platform and driver vendor provide them (mobiles: phone vendor provides) • can be hardware-accelerated • can be specific to platform features e.g. VR headset (spatializer) • can be controlled by parameters, but no exact/precise audio outputs • no GUI • consumers: recorder, video and voice calls, streaming apps etc. (not very suitable for DAWs) e.g. android.media.audiofx API for apps, AOSP Audio effect HAL for devices Trend -
Open Audio Plugin Formats with common interface, any host can
load any plugin Popular formats: • Steinberg VST3 • Apple AudioUnit • Linux community LV2 • Bitwig CLAP No one can break the ABI Music software always drives audio plugins ecosystem -
Open Audio Plugin Formats without plugin format, you can still
ship a DAW, but... -
Open Audio Plugin Formats without open plugin formats: you can
still ship a DAW, but... "The plug-ins Steinberg offer do not compare to the plethora of VSTs, AUs that iOS users can install. This needs to change in all fairness to make it a complete DAW." "Unfortunately the given limitations on Android, prevents the app to show up with the same features compared to iOS" (more of those voices at FL Studio Mobile reviews) -
Cross-Platform development • VST3, LV2, and CLAP are all cross-platform
(except for UI) • Users want plugins available on multiple platforms + multiple plugin formats ◦ in consistent behavior • Plugins are (usually) native apps based on C ABI (application binary interface) -
Cross-Platform development • VST3, LV2, and CLAP are all cross-platform
(except for UI) • Users want plugins available on multiple platforms + multiple plugin formats ◦ in consistent behavior • Plugins are (usually) native apps based on C ABI (application binary interface) • developers tend to write cross-platform code for audio plugins (in C++) ◦ using JUCE, DPF, iPlug2 etc. -or- using clap-wrapper ("CLAP first") • many of existing plugin sources are cross-platform code using ^ Trend -
Desktop vs. Mobile: totally different ecosystems Do we run desktop
software on mobiles? No... -
Desktop vs. Mobile: totally different ecosystems Desktop Mobile how we
run apps many windows show plugin within DAW one app on entire screen remote UI is restricted screen size big, show everything small, cannot show all load and save files usual awkward input events mouse + keyboard touches -
Desktop vs. Mobile: totally different ecosystems Desktop Mobile app installation
sloppy: download from anywhere app can load third-party libs strict: usually via app stores each app has user context how plugins are used load as a library connect as a different app JIT compilation possible prohibited on iOS standard plugin formats a few options only AUv3 on iOS nothing on Android (On macOS, Mac App Store apps are similarly strict) -
What should we achieve on "mobile version" ? • Integration
(interoperable) with desktop apps • build and run subset of desktop version ◦ not to achieve the same workflow, but make project data round-trip (shareable with desktop) example: Logic Pro -
How Apple ecosystem works • consistent API with desktop (CoreAudio,
CoreMIDI, AudioToolbox etc.) • audio plugin format exists: AudioUnit v3 ◦ AUv3 works on both macOS and iOS ◦ interoperable with AUv2 (desktop in-process only) : migration without rewrite • with audio plugins, audio apps gain more features It is not loved by desktop devs, but on secure platform everyone needs to behave. -
AUv3 technical aspects Works well with iOS app model •
based on App Extension • privileged IPC on real-time audio thread using AudioWorkgroup -
AUv3 technical aspects UI App Extension • host can launch
plugin UI • has "compact view" that shows small UI -
AUv3 technical aspects Smooth migration for AUv2 developers • AudioToolbox
provides consistent hosting API between AUv2 and AUv3 • AUv2 plugin can be loaded as AUv3 using AUAudioUnitV2Bridge -
Apple: tight MIDI integration MIDI is integrated from driver level
(CoreMIDI) to AudioUnits. The earliest party to have adopted MIDI 2.0. Internally CoreMIDI is based on MIDI 2.0, downtranslates to 1.0 only if needed. Still not complete, but the most advanced MIDI platform. -
An experimental audio plugin format for Android Audio apps should
be ubiquitously available None of existing formats works well with these constraints on Android There should be something comparable to AUv3 -
Crossing app border (IPC) • Android has fairly good Binder
IPC with real-time priority inheritance • but not exposed to framework binder domain • there is nothing like Audio Workgroup -
GUI Options Option 1: SurfaceControlViewHost embed UI from another app
(using SurfaceView) It is doable because host app instantiates them (security risk otherwise) Option 2: Web UI loaded by host WebView (there may be communication limitations) -
Reuse existing code If we create a new plugin format,
will people write code for it? => Very unlikely 😥 Chicken and Egg - no plugins, then no hosts. No hosts, then no plugins Why not reuse existing plugins then! -
Reuse existing code This is how AAP achieved those bunch
of plugins: • aap-juce build JUCE plugins (and hosts) as AAP • aap-lv2 wrap LV2 plugins (of non-JUCE plugins) as AAP In the future, CLAP wrapper is also doable (when many CLAP API based OSS plugins show up) -
Reuse existing code Not all features are imported • LV2
UI does not support Android • JUCE UI does not work as native UI (yet) ◦ it likely needs significant rewrite of JUCE IMO there should be different UI for mobiles Ideally this should work... -
Some design details AAP makes full use of MIDI 2.0
UMP • Events (note-on/off etc.) are fully based on UMP • Parameter changes are MIDI 2.0 Assignable Controllers (NRPNs) • Supports per-note expression and pitchbend • Extension API is built on top of SysEx messages • Future: state binaries and param./prog. metadata in MIDI-CI for sharing with potential MIDI-CI compatible desktop plugins AudioUnit supports UMP as well (parameters API is different) JUCE 9 (next major ver.) will support UMP in its plugin API too Trend -
Alternative Ideas WebAssembly DSP + WASM runtime + Web UI
? (assuming we can JIT compile) • pros can run DSP in process without privilege inheritance host can be simple WebView • cons WASM may not perform well interop between DSP and WebView in separate process can be tricky less possibility to port existing plugins (must be emscripten-compatible) -
Wrap-up Many audio apps potentially make use of audio effects
Bringing in audio plugins ecosystem to mobile is hard (app process model, UI) Port existing plugin apps to build up our own plugin API ecosystem Make use of open technology to make things easier and useful