Android and Audio Plugins: State of Union

ADC Japan 2026

Transcript

1. Android and Audio Plugins: State of Union atsushieno Slides at https://speakerdeck.com/atsushieno
2. Introduction
3. Android is missing common audio plugin formats Windows / Mac / Linux VST3, LV2, CLAP Mac - AUv2, AUv3 iOS AUv3 Web WebCLAP? WAM? Android 😭
4. Public plugin formats bring in more values on music apps
5. AAP: Audio Plugins For Android (why I am here to talk about it) An experimental plugin format and implementation that is designed to work on Android platform, which - users can combine any DAW with any plugin from different app packages - Your favorite plugin vendor does not have to "partner" with your DAW vendor (and vice versa) - can show plugin UI (Compose, JUCE, Web...) - some plugins and hosts ported from LV2 (aap-lv2) and JUCE (aap-juce)
6. What we discuss through this session 󰡈 find what kind of features we could achieve on Android 󰡈 how we can realize these features 󰢃 how to use AAP (*my* plugin format), AAP API details 🙆 what AAP ecosystem does NOT provide so far
7. This is not a typical Android audio dev. talk Typical Android audio dev. talk is about • how to build C++ app using Android NDK with Java/Kotlin • how to use Oboe • how to integrate MIDI (Android MIDI API or AMidi NDK API) • how to build Android app UI based on { Android view / Jetpack Compose } This session does not cover any of above(!) See DonTurner-ADC18 on those topics.
8. Demo
9. Video recordings In case I failed hard😅 https://youtu.be/OXSoqKCzbK4 The host app is atsushieno/uapmd
10. Designing a Plugin Format
11. Audio Plugin Format is an Interface (or Protocol or alike) Plugin Format itself is NOT an SDK 1. Its first priority is to preserve ABI compatibility. 2. Ease of development is just a nice to have. Low priority. 3. It must be kept in C ABI or more generic level, with RT safety. 4. The scope of the API should be limited to interaction between host and plugin Should a plugin format API specify messaging between plugin DSP and plugin UI? LV2: YES / CLAP: NO (cf. WebCLAP?)
12. Who should design a plugin format? • Apple owns AudioUnit ⇛ Google then? Can this be open? • Steinberg owns VST3 and Bitwig owns CLAP ⇛ DAW vendors then? Any of those on Android? ◦ actually there was: AAP is NOT the first attempt for audio plugin format. n-track AudioRoute SDK (~2021) • In principle, anyone should be able to build ones
13. Can't we reuse existing formats? VST3? CLAP? LV2? Not likely... They are all designed for desktop. We have to build new ones anyway. We will visit this later (once we all understand Android-specific issues).
14. Can't we use JUCE as the plugin format? No. • JUCE API is in C++, expecting static linking, which does not preserve ABI. • JUCE is a plugin/host SDK-or-framework (among many features). C++ for ABI does not work. (We learned the lesson from VCVRack.)
15. 🤖🤖🤖 Android 🤖🤖🤖
16. Cross process architecture
17. IPC instead of dlopen() Desktop: DAWs load plugin DLLs. Modern mobile platforms have isolated app setup. We cannot dlopen() plugin *.so libs. (restriction by Google Play Store) => use IPC instead
18. Binder IPC and realtime safety Service + NdkBinder for IPC is the lowest level down to kernel level Binder itself is capable of realtime priority inheritance, but it is unavailable to app devs... (Binder calls from audio thread indeed does not inherit priority.) cf. iOS AudioUnit v3 audio processors work in RT-safe manner (XNU Turnstiles) https://source.android.com/docs/core/architecture/hidl/binder-ipc
19. Realtime Interactions between Host and Plugin • Even if IPC can be performed in RT-safe manner, interaction between host and plugin should be kept minimum in audio processing. There will be a lot of context switches. • Callbacks from plugin to host within audio processing should be avoided too. ◦ CLAP thread-pool extension (microthreading) ◦ AUv2 HostCallback_GetBeatAndTempo() etc.
20. GUI
21. What plugin GUI is for? (It is more than just GUI...) • plugin-developer-defined controllers that may not fit well with knob or slider • properties, non-float parameters ◦ LV2 supports non-float parameters (Atoms), but typical multi-format DAWs and plugins don't support them
22. What plugin GUI is for? (It is more than just GUI...) • unmanaged presets by the plugin format API Most of existing plugins don't expose presets via presets API (!) Even though GUI is not mandatory, plugins without GUI drops a lot...
23. Plugin UI architecture options on Android • SurfaceControlViewHost - almost good, but still problematic ◦ IME does not work with SVCH as far as I tried (it matters in every language, not just CJKV) • Web UI - not much less ability, but needs JS-DSP interaction • Notification Bubbles - awkward displaying, but IME would work ("chat sample") • system alert window - deprecated platform feature, hard to manage AAP supports SCVH and Web UI in principle
24. dealing with GUI scaling and resizing How do we deal with resizable plugin UI? • show entire content: it's very often too small on mobile screens
25. dealing with GUI scaling and resizing How do we deal with resizable plugin UI? • partially: does that match what plugin devs want? annoying, but still usable
26. JUCE GUI integration pitfalls JUCE plugins and JUCE itself often expect one global display (`addToDesktop()`) - PopupMenu etc. JUCE dialogs sync API is unavailable on Android JUCE expects that a launcher Activity within the app itself (not true for Service). aap-juce comes up with a handful of JUCE patches for handful of versions🙃
27. WebView UI? Web UI would be portable. WebView interop hosts aren't... iPlug2, DPF(-webui): does not support Android Cmajor: choc WebView does not support Android Commonizing with WebCLAP can be an option.
28. Filesystem vs. Document Provider
29. Filesystems We have no shared "home" on Android. We use per-app data storage. We have to use platform Document Provider API / Storage Access Framework. Where I was stuck without files: - Presets - Sampler files - DSP scripts (e.g. JSFX in aap-juce-ysfx) They are bundled as assets, extracted to local filesDir at first run.
30. Threading Requirements
31. Asynchronous Plugin API (still needs more work) AAP internals avoid GUI thread requirement + synchronous API etc. - currently state API itself is sync, but the SDK part uses it async way. - expecting synchronous execution over IPC is error-prone: Fabian Renn-ADC24 - synchronous state API on main thread can block more often cf. clap issue #484 (asynchronous state API) I'm quite unsure if VST3 and CLAP APIs are safe to reuse. (JUCE plugins seem to be working though.)
32. Further Thoughts
33. Not Custom {VST3 / LV2 / CLAP} on Android? • VST3 and CLAP require too much main thread work that tend to block. • LV2 and CLAP require custom UI type. (VST3 too but the API is abstract) ◦ We don't go GUI-less. GUI matters (for good or not). • WebCLAP "Wasm as common ABI" is an interesting solution ◦ actually even Android NDK team once thought of that (ndk/issues/1771) ◦ I might add dual DSP bundle (WASM for host, native for plugin) in the future. • adopting WebCLAP / WebView UI practically means GUI-less nowaday. ◦ Can we integrate juce_emscripten? For now I find AAP defining its own API is more practical.
34. How AAP deals with extensibility AAP defines only very primitive calls on Binder IPC: • create(), prepare(), activate(), deactivate(), process(), extension() • when they change, that means essential format change (like VST2 -> VST3) AAP extensions (states, presets, UI, etc.) are designed as binary protocol Transport is either Binder (non-RT) or MIDI 2.0 SysEx on process() (RT). AAP State extension can work like Get and Set Device State MIDI-CI Properties.
35. AAP is almost feature complete, but misses... • API that is Polished and Stable ◦ Plugins: Build LV2 plugin or JUCE plugin and then port to Android ◦ Hosts: Use aap-juce (aap_audio_plugin_client) or atsushieno/uapmd (still not as easy as clap-juce-extensions) • Really useful DAW kind of hosts ◦ atsushieno/uapmd on Android: confirmed 6 tracks without glitches with 1024 bytes buffer. (uapmd is primarily a desktop plugin host and sequencer...) • Released Apps on Play Store ◦ Use atsushieno/android-ci-package-installer or Obtainium ◦ Hopefully release soon-ish • Business Plans💸
36. Ending...
37. Summary • We need plugin formats that respect Android as the first citizen. ◦ AAP is the only viable one so far. • Android has its own difficulty to achieve plugin features in: ◦ cross process architecture (namely it is not RT-safe) ◦ GUI ◦ file systems ◦ asynchronous API If you are to build a plugin format for Android, you will be hit by these issues😈
38. Thanks! Please have a look at atsushieno/aap-core for further details. Music Tech Meetup (Tokyo, sometimes online) https://music-tech.connpass.com/