Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unraveling the Internals of AppFunctions

hiro
November 13, 2026
9

Unraveling the Internals of AppFunctions

Kotlin Fest 2026 presentation.

hiro

November 13, 2026

Transcript

75 slides
  1. About me Nosho Hiroaki (hiroaki404) Android engineer / 6–8 years

    of Kotlin Kotlin Fest 2025: "Getting Started with AI Agent Development using Koog" gentrans: a CLI translation tool built with Koog X: @hiroaki08181359 Plugin for IntelliJ-based IDEs Markdown translation / Local LLM support github.com/hiroaki404/gentrans 2
  2. What are AppFunctions? A mechanism on Android that lets AI

    agents discover and execute functions in various apps AIagent Similar to MCP or function calling ƒcallfunctions Gemini and others can use your app as a tool Officially announced as Android MCP at Android Dev Summit, October 2026 Notesapp developer.android.com/ai/appfunctions / The Android Show: I/O Edition 2026 / Android Dev Summit 2026 Mapapp Calendarapp 3
  3. Today's session Follow the internals of AppFunctions through the source

    code androidx and AOSP Build our own agent app that uses AppFunctions with Koog JetBrains' AI agent framework Motivation To be able to design good tools Technical curiosity To learn how AI agents on mobile devices are built with Kotlin For how to add AppFunctions to your own app and best practices, see the official guide 5
  4. Agent and Tool Agent Agent The agent app that calls

    functions AIagent e.g. Gemini Tool ƒcallfunctions The app that exposes functions e.g. your app Notesapp Mapapp Tool Calendarapp Usually we mostly care about the Tool side. Today we also look inside the Agent side 6
  5. Code developers write on the Tool side /** * Books

    a flight ... * @param fromCity The departure city. ... * @param toCity The arrival city, ... * @param date The departure date. * @return The confirmed booking, ... */ @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun bookFlight( fromCity: String, toCity: String, date: LocalDate, ): FlightBooking = ... Function description passed to the agent Function declaration (Manifest and XML configuration are also required) With the official android skills, setup and writing functions are easy Demo: tool/tripapp/src/main/java/.../BaseTripAppFunctionService.kt (excerpt) 7
  6. Current limitations Agent: requires EXECUTE_APP_FUNCTIONS Regular apps cannot hold it

    Only allowed apps such as Gemini can use it * This does not apply to an in-app agent calling its own app's functions Tool: anyone can write @AppFunctionDeclaration Until now: only EAP (Early Access Program) partners were called from Gemini Since late October: Connected Apps Platform (preview) started. Published to Gemini after Play review (Currently only US developers who applied and were approved) developer.android.com/ai/appfunctions / issuetracker b/490650368 #15 / Create and manage connected apps / Android Dev Summit 2026 8
  7. How to test Try a Tool Check registration and execute

    with cmd app_function Call it from the official Testing Agent Build an Agent (2 ways to get the permission) ① Rooted emulator/device ← demo ② am instrument (Testing Agent) # List registered functions adb shell cmd app_function list-app-functions \ --package <package name> # Execute a function adb shell cmd app_function execute-app-function \ --package <package name> --function '<function ID>' \ --parameters '<JSON>' # priv-app approach (today's demo) emulator -avd Pixel_10_Root -writable-system adb root && adb remount adb push privapp-permissions-....xml \ /system/etc/permissions/ adb push agent.apk /system/priv-app/ adb reboot Details (Japanese): https://qiita.com/hiro404/items/338300798ca31e7bef87 github.com/android/appfunctions (Testing Agent) 9
  8. Versions and caveats Version androidx.appfunctions 1.0.0-beta01 Device Android 17 (API

    37) emulator AOSP (system-side source) android17-release (Android 17) Caveats Today's content reflects the current state. It may change This session is based on my own investigation of the code and my own views. It is not an official position. Some parts are simplified. 10
  9. Three questions 1. Developers only write an annotation and KDoc.

    How can the agent find and choose functions? 2. How can it find apps installed after the agent app started? 3. For a function in another process, who authorizes it, and how does it run? 12
  10. The big picture Worker Cataloger Toolapp ƒ Catalog Registerfunctions Oninstall

    Functionlist Register dothework Checkcatalog Request Frontdesk Relayexecution Result Client Agent 13
  11. Architecture Buildtime Toolapp @AppFunctionDeclaration(KDoc)/EntryPoint KSP(appfunctions-compiler) GeneratedKotlin GeneratedXML Service/Inventory Goesintoassets Oninstall/unlock

    )llatsnita(KPAnideldnuB Writtenbydeveloper system_server Runtime PackageManager Listintent-filters/getProperty AppFunctionManagerService MetadataSyncAdapter enabledstate Runtime AppFunctionService onExecuteFunction(main) Binder Runtime @AppFunctionDeclarationfunctionbody ParseXML androidx(Kotlin) Binder(observe) Binder(search) PlatformAppFunctionReader Search/observeviaOSManager Staticmetadata apps-db AppFunctionManagerServiceImpl UIDmatch/permission/enabled Binder frameworkAppFunctionManager AttachcallingPackage ServiceHelper/RemoteServiceCaller Dispatchwithwhen Developer AppFunctionManager observe/executeAppFunction AppsIndexer AppSearchManagerService JoinSpec appfunctions-db Agentapp resolve→bindeachtime framework AppSearch(APEX) Binder(4calls) In-process Viafiles By the author. Versions used: androidx appfunctions 1.0.0-beta01 / AOSP android17-release / Demo: github.com/hiroaki404/appfunctions-koog-agent-demo 15
  12. ①–③ Registration flow Buildtime Tool Toolapp Atinstall Runtime ①KSP ⑧Functionbody

    GenerateXML system_server OS Agent Agentapp Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution KSP generates XML AppsIndexer reads the XML and registers the functions in the AppSearch database 16
  13. ④–⑧ Discovery and execution flow Buildtime Tool Toolapp system_server OS

    Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Agent Agentapp Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution The agent gets the list from AppSearch and observes changes with a Flow Agent and Tool never connect directly 17
  14. Tool The side that exposes functions Buildtime Tool Toolapp system_server

    OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 18
  15. Functions of the demo tool apps The travel app and

    messaging app from the opening demo They expose 5 functions App Function tripapp getTravelInfo Returns recommended spots tripapp saveItinerary Saves an itinerary tripapp searchItineraries Searches itineraries incl. past trips tripapp bookFlight Books a flight → Return String, Int → List<Spot> Itinerary → String String? → List<Itinerary> String, String, LocalDate → message sendMessage Sends a message to a contact String, String What it does github.com/hiroaki404/appfunctions-koog-agent-demo (tool/) Args FlightBooking → String 19
  16. What developers do 1. build.gradle.kts : add the appfunctions dependency

    and KSP 2. Write @AppFunctionDeclaration in an abstract Service annotated with @AppFunctionServiceEntryPoint (KDoc supported) 3. res/xml/app_metadata.xml : describe the whole app 4. AndroidManifest.xml : declare the generated Service and XML 20
  17. ① Gradle: add dependencies plugins { id("com.google.devtools.ksp") version "2.3.10" }

    dependencies { Add the appfunctions and KSP dependencies implementation("androidx.appfunctions:appfunctions:1.0.0-beta01") ksp("androidx.appfunctions:appfunctions-compiler:1.0.0-beta01") } Demo: tool/tripapp/build.gradle.kts / tool/gradle/libs.versions.toml 21
  18. ② Write functions: EntryPoint and @AppFunctionDeclaration @AppFunctionServiceEntryPoint( serviceName = "TripAppFunctionService",

    appFunctionXmlFileName = "trip_functions", ) abstract class BaseTripAppFunctionService : AppFunctionService() { /** * Books a flight ... * @param fromCity The departure city. ... * @param toCity The arrival city, ... * @param date The departure date. * @return The confirmed booking, ... */ @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun bookFlight( fromCity: String, toCity: String, date: LocalDate, ): FlightBooking = ... Name of the generated concrete class Name of the generated XML file Function description for the LLM Use KDoc as the description You can declare multiple functions @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun saveItinerary(itinerary: Itinerary): String = ... } Demo: tool/tripapp/src/main/java/.../BaseTripAppFunctionService.kt 22
  19. ③ app_metadata.xml: describing the whole app Describes the whole app

    (not each function) The first entry point in the official guide Provided by the developer in res/xml/ <!-- res/xml/app_metadata.xml --> <AppFunctionAppMetadata xmlns:appfn= "http://schemas.android.com/apk/androidx.appfunctions" appfn:instructions="Travel app ... Whole-app description for the user's trips: searching saved itineraries, ... and booking flights. Operational Patterns: - To plan a new trip, call getTravelInfo ..., then pass the chosen spots ... to saveItinerary. - Flight numbers ... come from the bookFlight result. Constraints: - Dates written as text ... use yyyy-MM-dd ..." appfn:displayDescription= "@string/app_function_app_display_description" /> Demo: tool/tripapp/src/main/res/xml/app_metadata.xml (excerpt, reformatted) 23
  20. ④ Manifest: name the generated files <property> points to app_metadata.xml

    Match serviceName / appFunctionXmlFileName with the <!-- AndroidManifest.xml (directly under <application>) --> <property android:name="android.app.appfunctions.app_metadata" android:resource= app_metadata.xml "@xml/app_metadata" /> <service Concrete Service name android:name=".TripAppFunctionService" Only the system can bind values above android:permission= With BIND_APP_FUNCTION_SERVICE , only the system can bind (required; it is a condition the system uses when searching) android:exported="true"> "android.permission.BIND_APP_FUNCTION_SERVICE" <property android:name="android.app.appfunctions.v2" android:value="trip_functions.xml" /> <property android:name="android.app.appfunctions.schema" android:value="app_functions_schema.xsd" /> <intent-filter> <action android:name= "android.app.appfunctions.AppFunctionService" /> </intent-filter> </service> Demo: tool/tripapp/src/main/AndroidManifest.xml 24
  21. File generation with KSP KSP (Kotlin Symbol Processing): interprets code

    and generates files Nextroundalsoreadsoutputs kspDebugKotlin(KSP) Source .kt Symbols KSPparsessource KSPprocessor Resolveneededtypes ReadKDoc,writefiles Generatedfiles .kt / .xml etc. Maincompileafterlastround compileDebugKotlin (main compile) Source +generatedfiles FIR Parse IR ACompilerPlugin Canrewritethis Bytecode KSP parses the source into symbols, so types, fields, methods, KDoc, etc. can be read. It never rewrites existing code; it only generates files 25
  22. KSP entry point: AppFunctionCompiler The entry is AppFunctionCompiler, created by

    a Provider registered in META-INF/services In the round where the required types are resolved, it runs the child processors (9 of them) in order ServiceEntryPoint Inventory // AppFunctionCompiler.kt class AppFunctionCompiler( private val processors: List<SymbolProcessor>, private val logger: KSPLogger, ) : SymbolProcessor { override fun process(resolver: Resolver) : List<KSAnnotated> { val deferred = shouldDeferAllProcessing(resolver) if (deferred.isNotEmpty()) { deferred } else { buildList { for (processor in processors) { Call child processors in order addAll(processor.process(resolver)) Serializable } } } ... androidx: appfunctions-compiler/.../AppFunctionCompiler.kt L44-64 (try/catch omitted) 26
  23. AppFunctionServiceEntryPointProcessor Collects classes annotated with @AppFunctionServiceEntryPoint Also gets each function's

    KDoc via docString Generates the concrete Service and the XML The XML is written with createNewFile using the xml extension // AppFunctionServiceEntryPointProcessor.kt override fun process(resolver: Resolver) : List<KSAnnotated> { ... val serviceEntryPoints = appFunctionSymbolResolver .resolveAnnotatedAppFunctionServiceEntryPoints() for (serviceEntryPoint in serviceEntryPoints) { generateAppFunctionService(serviceEntryPoint) generateXml(serviceEntryPoint, ...) } return emptyList() } Generate files // AppFunctionSymbolResolver.kt (collecting side) resolver.getSymbolsWithAnnotation(...) ... appFunctionDeclarations.map { AnnotatedAppFunction(it, it.docString) // KDoc } androidx: processors/AppFunctionServiceEntryPointProcessor.kt L101-122 / core/AppFunctionSymbolResolver.kt L37-63 (excerpt; try/catch and args omitted) 27
  24. Generated files Concrete Service build/generated/ksp/debug/ Inventory ├── kotlin/dev/hiroaki404/.../tripapp/ │ ├──

    TripAppFunctionService.kt Factory │ ├── $BaseTripAppFunctionService_ │ │ XML │ └── $SpotFactory.kt ... ×4 AppFunctionInventory.kt ├── kotlin/appfunctions_aggregated_deps/ ← Concrete Service ← Inventory ← Factory (index for aggregation ×5) └── resources/assets/ └── trip_functions.xml ← XML Function descriptions 28
  25. Generated file ① Concrete Service Purpose Receives system calls and

    routes them to the developer's function Contents onExecuteFunction gets the Request and metadata Branches on the ID with when and calls bookFlight IDs are const s in the class Args are cast from a Map Demo: generated code (excerpt) public class TripAppFunctionService : BaseTripAppFunctionService() { override fun onExecuteFunction( request, metadata, ...) = AppFunctionExecutionDispatcher .dispatchExecuteAppFunction( scope, request, metadata, ... ) { parameters -> when (request.functionIdentifier) { FUNCTION_ID_BOOK_FLIGHT -> this.bookFlight( parameters["fromCity"] as String, parameters["toCity"] as String, parameters["date"] as LocalDate) ... } } companion object { const val FUNCTION_ID_BOOK_FLIGHT = "...BaseTripAppFunctionService#bookFlight" } } Receives the Request By ID Type cast Function ID 29
  26. Generated file ② Inventory: function metadata Purpose Holds the information

    to convert the Request received by the Service into the defined argument types, and to convert the return value into the form to send back // $BaseTripAppFunctionService // _AppFunctionInventory.kt (generated) override val functionIdToMetadataMap = mapOf( "...BaseTripAppFunctionService#bookFlight" to ...BookFlightMetadataObject .APP_FUNCTION_METADATA, ... ) private object ...BookFlightMetadataObject { private val FROMCITY_PARAMETER_METADATA = Contents AppFunctionParameterMetadata( name = "fromCity", isRequired = true, Function metadata and type information dataType = ...FROMCITY_PRIMITIVE_DATA_TYPE, description = "The departure ...", A Map from function ID to metadata Built from the defined function's types, KDoc, etc. ID → metadata From KDoc ) ... val APP_FUNCTION_METADATA = ... } Demo: build/generated/ksp/debug/kotlin/.../$BaseTripAppFunctionService_AppFunctionInventory.kt (excerpt, 535 lines) 30
  27. Generated file ③ XML: registered in the function list Purpose

    Read by the system to register functions in AppSearch Contents Nearly the same metadata as the Inventory Descriptions come from KDoc (only the XML has them) Built by the ServiceEntryPoint processor with Java's DOM XML API Goes into the APK's assets/ <appfunctions> <appfunction> <id> ...BaseTripAppFunctionService#bookFlight </id> <enabledByDefault>true</enabledByDefault> <description> Books a flight ... </description> <parameters> <dataTypeMetadata> <isNullable>false</isNullable> <type>8</type> </dataTypeMetadata> <isRequired>true</isRequired> <name>fromCity</name> <description> The departure city. ... </description> </parameters> ... Demo: build/generated/ksp/debug/resources/assets/trip_functions.xml (excerpt, about 18KB) From KDoc 31
  28. From KDoc to LLM-facing descriptions Inventory KDoc /** ... */

    KSP docString Justextractsastring androidx DocstringUtils Splitstagsitself Readbyappatruntime Docsgoto2places XML ReadbyOSatinstall KSP passes just a string. @param is parsed by androidx with regular expressions androidx: appfunctions-compiler/.../core/DocstringUtils.kt L22-27 32
  29. Code generation: building code with KotlinPoet KotlinPoet: a library (by

    Square) that builds Kotlin source with Builders instead of string concatenation beginControlFlow( "when (%L.%L)", "request", "functionIdentifier") The concrete Service's when for (appFunction in The concrete Service and Inventory are generated with KotlinPoet serviceEntryPoint.appFunctions) { beginControlFlow("%L ->", getFunctionIdConstantPropertyName( appFunction)) add("this.%N(\n", function.simpleName.asString()) for (param in function.parameters) addStatement("%L[%S] as %T,", "parameters", paramName, param.type.toTypeName()) ... endControlFlow() Division of roles: KSP reads symbols and KDoc and provides the output files. KotlinPoet and Java's XML library build the contents Type cast } androidx: processors/AppFunctionServiceEntryPointProcessor.kt L284-355 (excerpt) 33
  30. Summary: who uses the generated files, when, and how Toolapp

    Buildtime Atinstall ConcreteService Tofunctionviawhen Inventory Restoreargtypes Factory Repackdataclasses Functionentrypoint KSP Runtime DispatchbyfunctionID Functionspec(Kotlin) Convertperspectypes Fordataclasses AppFunctionData⇄dataclass XML bind Functionspec(XML) system_server AppsIndexer RegisterXMLinAppSearch Whereagentsfindfunctions AppFunction ManagerService 34
  31. Registering with the system Until the XML lands in AppSearch

    (registered in the function list) Buildtime Tool Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 35
  32. From app install to registration system_server ToolAPK ①Install PackageManager ②ACTION_PACKAGE_ADDED

    OnlynotestheManifest ③QueryXMLpath Manifest assets/*.xml AppsIndexer Registerreceiveratboot ⑤put ④ReadXMLviaAssetManager AppSearch appfunctions-db apps-db Enabled/disabled ⑥Sync Functiondefinitions AppFunctionManagerService Notifychanges Watchapps-dbchanges AppsIndexer in system_server registers functions in AppSearch 36
  33. What is AppSearch? AppSearch is a high-performance on-device search solution

    for managing locally stored, structured data. An on-device full-text search engine + document store // Sample from the official guide (shortened) @Document data class Note( @Document.Namespace val namespace: String, @Document.Id val id: String, @Document.StringProperty( indexingType = INDEXING_TYPE_PREFIXES ) val text: String, ) session.setSchema( SetSchemaRequest.Builder() .addDocumentClasses(Note::class.java) .build()) session.put( PutDocumentsRequest.Builder() .addDocuments(note).build()) session.search("fruit", searchSpec) developer.android.com/develop/ui/views/search/appsearch (2026-10-11. Quote from the opening) 37
  34. AppSearch keywords Keyword Database Description in the official guide A

    collection of documents conforming to a schema. Only the creating app (system) can open it Session The entry point for operations, returned when opening a DB Document A unit of data. Uniquely identified by Namespace and ID Schema The structure of the data Namespace Used to separate data from different sources * In AppFunctions, one function corresponds to one Document Create Put Search OpenDB,setschema PutDocuments Searchbyquery createSearchSession put search developer.android.com/develop/ui/views/search/appsearch (Database and session / Schema and schema types / Documents) 38
  35. Storage and Session types Storage Where the index lives Shared

    with other apps LocalStorage The app's data directory No PlatformStorage system_server (Android 12+) Yes PlayServicesStorage Google Play services Yes Session Connects to AppSearchSession One DB opened by name GlobalSearchSession Searches across all DBs it has access to The AppFunctions DB is in PlatformStorage developer.android.com/develop/ui/views/search/appsearch (Platform Storage, Local Storage or Play Services Storage, 2026-10-07) / androidx appsearch: Javadoc of AppSearchSession.java, GlobalSearchSession.java, PlatformStorage.java 39
  36. AppsIndexer writes to AppSearch The AppSearch module has separate indexers

    per data category AppsIndexer operates the AppSearch DB (apps-db) for AppFunctions AppsIndexer defines the Schema and puts Documents packages/modules/AppSearch/service/.../appsearch/ ├── contactsindexer/ Contacts ├── appsindexer/ App list + AppFunctions └── indexer/ Common periodic maintenance The package contains the implementation of the AppsIndexer used to index metadata about apps and app functions exposed by apps into AppSearch. AOSP AppSearch: appsindexer/package-info.java 40
  37. AppsIndexer watches app installs At boot, system_server starts the AppSearch

    module, and AppsIndexer starts watching Three triggers // AppsIndexerManagerService.java private void registerReceivers() { IntentFilter appChangedFilter = new IntentFilter(); appChangedFilter.addAction( Intent.ACTION_PACKAGE_CHANGED); Package changes (add, update, remove) appChangedFilter.addAction( First unlock / every 30 days appChangedFilter.addAction( On package-change notifications from PackageManager, it re-checks all packages Intent.ACTION_PACKAGE_ADDED); Install Intent.ACTION_PACKAGE_REPLACED); appChangedFilter.addAction( Intent.ACTION_PACKAGE_FULLY_REMOVED); appChangedFilter.addDataScheme("package"); mContext.registerReceiverForAllUsers( new AppsProviderChangedReceiver(), appChangedFilter, ...); Register receiver AOSP AppSearch: AppSearchModule.java L146-150 / appsindexer/AppsIndexerManagerService.java L132-149, L227-236 / AppsIndexerConfig.java L33 41
  38. AppsIndexer gets the XML file from the APK 1. Enumerate

    apps with an AppFunctionService via intent-filter (Services without BIND_APP_FUNCTION_SERVICE are excluded) 2. Get the XML path from the Service's <property> Open the XML at that path and read it with parseIntoMapForGivenSchemas // AppsUtil.java Intent appFunctionServiceIntent = new Intent(APP_FUNCTION_SERVICE_INTERFACE); List<ResolveInfo> services = packageManager .queryIntentServices(appFunctionServiceIntent, 0); ... // AppFunctionResolveInfo.java v2XmlProperty = packageManager.getProperty( APP_FUNCTION_V2_XML_PROPERTY_NAME, componentName); ... if (v2XmlProperty != null) { return v2XmlProperty; } ... return packageManager.getProperty( APP_FUNCTION_V1_XML_PROPERTY_NAME, componentName); ... // AppFunctionsIndexerUtil.java if (appFunctionXmlInfo.useSchemaForParsing()) { appFunctionXmlInfo.runWithXmlParser(packageManager, xmlParser -> packageAppFunctions.putAll( parser.parseIntoMapForGivenSchemas(...))); } AOSP AppSearch: AppsUtil.java L161-191 / AppFunctionResolveInfo.java L137-155 / AppFunctionsIndexerUtil.java L246-266 (excerpt) 42
  39. Opening the APK and reading the XML Creates an AssetManager

    from the Tool app's APK Uses AssetManager to open the APK's assets/ by path // AppFunctionResolveInfo.java (runWithXmlParser) Resources resources = packageManager.getResourcesForApplication( mPackageName); ... XmlPullParser parser = The KSP-generated trip_functions.xml is read here Each AppFunction (each <appfunction> tag in the XML) becomes one AppSearch document AOSP AppSearch: appsindexer/AppFunctionResolveInfo.java L456-477 (excerpt) XmlPullParserFactory.newInstance() .newPullParser(); try (InputStreamReader isr = new InputStreamReader( resources.getAssets() AssetManager .open(mXmlFile.getXmlFilePath()))) { parser.setInput(isr); action.accept(parser); } 43
  40. Two AppSearch DBs DB apps-db Contents Function definitions (static) Written

    by AppsIndexer Visibility Anyone who can see the Tool app Only the Tool app itself appfunctions-db Enabled/disabled (runtime) AppFunctionManagerService and apps with permissions such as EXECUTE_APP_FUNCTIONS Both are owned by android (system_server). Apps never write to them directly * Whether the Tool app is visible depends on the agent Manifest's <queries> / QUERY_ALL_PACKAGES (the demo uses the latter) AOSP AppSearch: AppSearchHelper.java L96, L284 / AOSP frameworks/base: AppFunctionRuntimeMetadata.java L49, MetadataSyncAdapter.java L305-320 44
  41. Agent search How the function list is fetched Buildtime Tool

    Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 45
  42. Observing and searching with AppFunctionManager For the Agent, androidx's AppFunctionManager

    is the entry point Observe: observeAppFunctions() Notifies only changes via a Flow Search: searchAppFunctions() Gets the list of functions The agent app combines the two // AppFunctionDiscovery.kt (demo app) fun AppFunctionManager.discoverAppFunctions() : Flow<List<AppFunctionMetadata>> = observeAppFunctions() Notifies only changes .filterIsInstance< AppFunctionsChangeEvent.MetadataChanged>() .map { } .onStart { emit(Unit) } // initial .map { Re-fetch the list searchAppFunctions(AppFunctionSearchSpec()) } Demo: AppFunctionDiscovery.kt (reformatted) / androidx: AppFunctionManager.kt L205-207, L237-241 46
  43. Down to AppSearch Both observing and searching go through an

    intermediary class and call the OS AppFunctionManager Agentprocess androidx AppFunctionManager Delegate androidx The OS one in PlatformAppFunctionReader android.app.appfunctions , separate from androidx From there, two destinations Observe → AppFunctionManagerService Search → AppSearch framework android.app.appfunctions.AppFunctionManager Observe Notify Search watch AppSearch system_server AppFunctionManagerService Notifychanges androidx: AppFunctionManager.kt L635-643 / AOSP frameworks/base: AppFunctionManager.java L621-657, L702-750 apps-db(definitions) 47
  44. Implementation of observeAppFunctions Inside a callbackFlow, registers an AppFunctionObserver with

    the OS AppFunctionManager Turns callbacks into events and calls trySend Metadata changes (per package) MetadataChanged State changes (per function) → → StatesChanged androidx: PlatformAppFunctionReader.kt L166-191 (excerpt, reformatted) // PlatformAppFunctionReader.kt callbackFlow { val appFunctionObserver = object : AppFunctionObserver { override fun onAppFunctionMetadataChanged( changedPackageNames: Set<String> ) { trySend(AppFunctionsChangeEvent .MetadataChanged(changedPackageNames)) } override fun onAppFunctionStatesChanged(...) { trySend(AppFunctionsChangeEvent .StatesChanged(...)) } } val observation: AppFunctionObservation = appFunctionManager.observeAppFunctions( Runnable::run, appFunctionObserver) awaitClose { observation.cancel() } }.buffer(Channel.UNLIMITED) callbackFlow Send event 48
  45. Implementation of searchAppFunctions Repacks the conditions into OS types package

    / function name / schemaName / schemaCategory / schemaVersion / scope The demo sets none functions → gets all visible No keyword filtering of descriptions Picking functions by meaning is the LLM's job suspendCancellableCoroutine: callback suspend → The OS searches AppSearch via GlobalSearchSession // PlatformAppFunctionReader.kt val platformSpec = searchFunctionSpec.toPlatformSearchSpec() return suspendCancellableCoroutine { cont -> appFunctionManager.searchAppFunctions( platformSpec, Runnable::run, ...) } // AppFunctionManagerHelper.java (OS) appSearchManager.createGlobalSearchSession( executor, sessionResult -> { GlobalSearchSession session = sessionResult.getResultValue(); ... SearchResults results = session.search( searchSpec .getStaticMetadataAppSearchQuery(), staticMetadataSearchSpec); }); androidx: PlatformAppFunctionReader.kt L62-103 / AOSP frameworks/base: AppFunctionManagerHelper.java L172-216 (both excerpts) 49
  46. Summary: registration to observing and searching system_server Install AppsIndexer put

    Agentprocess AppSearch Notifychanges AppFunctionManagerService NotifyAgentviaBinder Binder MetadataChanged observeAppFunctions Re-fetchlist searchAppFunctions The agent keeps observing. That's why apps installed later are found too 50
  47. Agent function execution How a function in another process runs

    Buildtime Tool Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 51
  48. The entry is executeAppFunction androidx's AppFunctionManager is the entry; it

    runs via executeAppFunction() Goes through the OS AppFunctionManager Before calling, androidx searches AppSearch once more To get the metadata used to convert the return type Then system_server binds and calls the Tool's Service Agentprocess ①Lookupmetadata androidx AppFunctionManager androidx ②Execute PlatformAppFunctionReader androidx PlatformAppFunctionManagerApi framework android.app.appfunctions.AppFunctionManager system_server AppSearch apps-db(definitions) ①Search ②Execute AppFunctionManagerServiceImpl Toolprocess androidx: AppFunctionManager.kt L127-153 / PlatformAppFunctionManagerApi.kt L199-251 / PlatformAppFunctionReader.kt L105-117 ③Bindandcall AppFunctionService Callfunctionbody 52
  49. system_server's pre-execution checks 1. The UID of the Agent's declared

    package name matches the actual UID 2. User checks (multi-user, work profile) 3. Execute permission 4. The function is enabled ( apps-db + appfunctions-db joined via JoinSpec) 5. Find the Tool's Service (from apps-db) Only Services with BIND_APP_FUNCTION_SERVICE (declared in the Manifest) AOSP frameworks/base: AppFunctionManagerServiceImpl.java L386-560, L652-760 53
  50. Execution permission checks Agent Has EXECUTE_APP_FUNCTIONS_SYSTEM Has EXECUTE_APP_FUNCTIONS No permission

    Result Always OK OK for its own functions. Others OK only if the Tool is on the allowlist OK for its own functions only (intended for ML Kit in-app agents) allowlist: per Agent (signature), the list of Tools it may call The list is queried from a provider outside the system The Tool side can also restrict callers with @AppFunctionAccessLevel (SELF / SYSTEM / ANDROID_TRUSTED; default ANDROID_TRUSTED) Not just EXECUTE_APP_FUNCTIONS ; being on the allowlist is also required AOSP frameworks/base: CallerValidatorImpl.java L98-170 / allowlist/SystemAppFunctionAllowlistReader.java L126-186 54
  51. Binding the service bindService per call BIND_FOREGROUND_SERVICE gives priority on

    par with a Foreground Service * Only when the Agent has the permission // RemoteServiceCallerImpl.java public boolean bindAndRun() { boolean bindServiceResult = mContext.bindServiceAsUser( mIntent, this, mFlags, mUserHandle); Bind per call if (bindServiceResult) { ... mDirectServiceVulture = () -> { ... mCancellationSignal.cancel(); }; ... mCallerBinder.linkToDeath( mDirectServiceVulture, /* flags= */ 0); ... } ... } ... public void onCompleted() { AOSP frameworks/base: RemoteServiceCallerImpl.java L117-185 (excerpt) / AppFunctionManagerServiceImpl.java safeUnbind();L618-624 } Unbind when done 55
  52. From the Tool app's process to function execution 1. The

    framework's AppFunctionService checks the caller (is it system_server?) 2. Switches to the main thread 3. androidx's AppFunctionService looks up metadata off the main thread, then switches back 4. The KSP-generated concrete Service dispatches with when and calls the function body on the main thread Toolprocess:Binderthread Framework AppFunctionService(IAppFunctionService.Stub) Mainthread getMainExecutor androidx AppFunctionService Metadata GeneratedService onExecuteFunction when(functionIdentifier) Developer'sfunction AOSP frameworks/base: AppFunctionService.java L118-170 / androidx: AppFunctionService.kt L84-148 / Demo: generated Service 56
  53. The whole flow (again) Buildtime Tool Toolapp system_server OS Agent

    Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution Agent and Tool never connect directly. The system is always in between 57
  54. What I built tool: two tool apps tripapp: getTravelInfo /

    saveItinerary / searchItineraries / bookFlight message: sendMessage appfunctions-koog-agent-demo/ ├── tool/ │ ├── tripapp/ Travel app │ └── message/ Messaging app └── agent/ Agent app agent: an agent app built with Koog Input field, responses, and tool-call display Shows the number and list of available tools LLM: gemini-3.5-flash-lite The agent is placed in priv-app on a rooted emulator to get the execution permission github.com/hiroaki404/appfunctions-koog-agent-demo (Koog 1.2.0) hiroaki404/appfunctions-koog-agent-demo 59
  55. Normally, Koog tools are fixed at build time Tools are

    written as classes Argument types are also fixed at build time But AppFunctions are only known at runtime Some apps are installed later // The usual way: a fixed tool that can only call bookFlight object BookFlightTool : SimpleTool<BookFlightTool.Args>() { @Serializable data class Args(val from: String, val to: String, val date: LocalDate) ... } val toolRegistry = ToolRegistry { tool(BookFlightTool) } Illustrative example (same idea as the demo's early fixed tool AddTool.kt, commit 6fe2519) 60
  56. Building tools from metadata From the AppFunctionMetadata obtained by search

    One function = one tool Build Koog's tool description, ToolDescriptor, at runtime The Tool's execute calls executeAppFunction Arguments are received as a JSONObject Same shape as Koog's McpTool Demo: agent/.../AppFunctionTool.kt / AgentRunner.kt (excerpt) Create one Tool per function // AgentRunner.kt val tools = appFunctions.mapNotNull { metadata -> val descriptor = metadata.toToolDescriptor() AppFunctionTool(manager, metadata, descriptor) } Build ToolDescriptor class AppFunctionTool( private val manager: AppFunctionManager, private val metadata: AppFunctionMetadata, descriptor: ToolDescriptor, ) : Tool<JSONObject, String>(..., descriptor) { override suspend fun execute( args: JSONObject, ): String { val parameters = ... // repack args val response = manager.executeAppFunction( ExecuteAppFunctionRequest( metadata.packageName, metadata.id, parameters)) return ... // response to JSON } } Args stay as JSON Run executeAppFunction 61
  57. Converting between Koog types and AppFunctions types // Inside toToolDescriptor()

    on p61: convert each arg's type fun AppFunctionMetadata.toToolDescriptor() = ToolDescriptor(name = ..., description = ..., requiredParameters = parameters.filter { ... }.map { ToolParameterDescriptor(it.name, ..., it.dataType.toToolParameterType(...)) }, ...) fun AppFunctionDataTypeMetadata.toToolParameterType( components: AppFunctionComponentsMetadata, ... ): ToolParameterType = when (this) { is AppFunctionStringTypeMetadata -> enumValues?.let { ToolParameterType.Enum(...) } ?: ToolParameterType.String is AppFunctionLongTypeMetadata -> ToolParameterType.Integer is AppFunctionReferenceTypeMetadata -> components.dataTypes[referenceDataType]!! .toToolParameterType(components, ...) ... } To Koog types ①Whenbuildingtools AppFunctionMetadata AppFunctionstypes ToolDescriptor Koogtypes ②Whenrunningtools Args(JSON) ReturnedbyLLM AppFunctionData ToexecuteAppFunction Follow references Demo: agent/.../AppFunctionSchemaConverter.kt (excerpt) 62
  58. What reading the design revealed What I learned The agent

    is an app holding a permission; the system lists and runs functions Tool apps and agent apps interact indirectly through the system It is built on Android's existing assets and Kotlin's, such as KSP, Flow, and KotlinPoet How what developers write is used Developers write When used How it gets there app_metadata.xml Runtime (if the agent reads it) Agent reads it directly via PackageManager KDoc / function definitions Registered at install; returned in search results KSP XML AppsIndexer AppSearch search results Service (implementation) When the function runs System binds and calls it → androidx: AppFunctionPackageMetadata.kt L115-130, AppFunctionSearchSpec.kt L95-125 → → What it affects Which app to use, and how → Function the LLM picks, and its args Processing, threads, errors, types 65
  59. How to design a good Tool Write with when app_metadata

    and KDoc are used in mind Put cross-function call order in app_metadata, and function descriptions in KDoc No need to stuff keywords for search, it seems (Depends on the agent implementation, though) Watch out for type complexity LLMs struggle with structured data. Test complex types especially well androidx: AppFunctionSearchSpec.kt L95-125 66
  60. How to design a good Tool (cont.) Concurrency and waiting

    Functions run on the main thread. Move heavy work off it with withContext ( Dispatchers.IO for I/O, Dispatchers.Default for computation) Timeouts are up to the agent. Hand long work to WorkManager and return "started" (e.g. downloads) Assume functions may run while the app is in use Some functions run only while the app shows a specific screen (@AppFunctionSignature) androidx: AppFunctionDeclaration.kt L26-30 / AOSP frameworks/base: AppFunctionService.java L128-135, ServiceConfig.java L47-50 67
  61. How to design a good Tool (cont.) Safety (confirming function

    execution) → The library has no confirmation mechanism generally don't expose destructive operations; if you do, ensure safety on the Tool side I prototyped a confirmation dialog and A2UI The Tool returns UI information, and the agent renders it with A2UI Approval parameters are defined by a custom agreement, so the agent can show a confirmation dialog AppFunctions probably has no such mechanism yet, so I hope for one in the future Demo: hiroaki404/appfunctions-koog-agent-demo (a2ui branch) 68
  62. Good Tool design: other points Don't expose domain-layer or data-layer

    methods as-is From the agent's view, it is the API layer Design inputs and outputs for agents, and add logic that assumes calls from agents Descriptions are read by the LLM. Write what the function does, not commands English is recommended (Google's answer) Arguments with default values become optional. But the values aren't used; null/0/false arrive Return errors as the AppFunctionException subclass matching the cause For details, see the official AppFunctions quality guidelines or the skill developer.android.com/ai/appfunctions/quality-guidelines github.com/android/skills/tree/main/device-ai/appfunctions Issue Tracker: b/551999767 (KDoc language) / AppFunctions Quality guidelines (2026-10-28) 69
  63. The future of AppFunctions Restrictions and permissions Until now, both

    Agent and Tool were limited to select apps Tool: likely to become widely available Agent: calling other apps remains privileged Own functions via ML Kit in-app agents When tools multiply, which gets chosen? androidx's AppFunctionManager cannot take keywords as a query It assumes fetching the whole list at once without filtering by content (Filtering by package name or function name is possible) The system does no ranking. Selection is up to the agent implementation and the LLM Like the situation with countless MCP servers and skills AOSP frameworks/base: AndroidManifest.xml L9492-9515 / SystemAppFunctionAllowlistReader.java L126-187 / ADS 2026 / d.android.com/distribute/aep 70
  64. The future of AppFunctions (cont.) Engagement Tasks get done without

    opening the app → what about ad and billing models? On the other hand, execution from agents creates a new inflow channel Functions can be enabled/disabled at runtime (e.g. only for paying users) Results can also lead users to open the app (PendingIntent) Security Prompt injection is often mitigated with guardrails on the agent side Still, Tools should be built without relying solely on the Agent or the system Don't expose whole DBs or files 71
  65. Summary Question Answer Q1. With just an annotation and KDoc,

    KSP generates XML, and AppsIndexer puts it into AppSearch. The how can it find and choose? LLM chooses after reading app_metadata and KDoc Q2. How can it find apps installed after the agent app started? Q3. For a function in another process, who authorizes it and how does it run? AppsIndexer registers them on PackageManager's install notifications. The change arrives via a Flow, and the agent searches again system_server checks it and mediates via Binder, running it in the Tool's process. Tools can also declare who may call them 72
  66. References (1/2) Official documentation AppFunctions overview (including FAQ) https://developer.android.com/ai/appfunctions Add

    AppFunctions to your app https://developer.android.com/ai/appfunctions/add-appfunctions AppFunctions quality guidelines https://developer.android.com/ai/appfunctions/quality-guidelines Create and manage connected apps https://developer.android.com/ai/appfunctions/create-manage-connected-apps AppSearch https://developer.android.com/develop/ui/views/search/appsearch androidx.appfunctions release notes https://developer.android.com/jetpack/androidx/releases/appfunctions Videos Android Dev Summit 2026 keynote https://www.youtube.com/watch?v=8Lmv7LGgL4E Integrate your app with the Android intelligence system https://www.youtube.com/watch?v=UsqDEfyOH2E Extend your app to Gemini and Search with the Connected Apps Platform https://www.youtube.com/watch?v=-4G-4w9FKDw The Android Show: I/O Edition 2026 https://www.youtube.com/watch?v=dXCCleAddEA 73
  67. References (2/2) Source code and samples androidx appfunctions (1.0.0-beta01) https://github.com/androidx/androidx/tree/androidx-main/appfunctions

    AOSP frameworks/base (Android 17) https://android.googlesource.com/platform/frameworks/base/+/refs/heads/android17-release/core/java/android/app/appfunctions/ https://android.googlesource.com/platform/frameworks/base/+/refs/heads/android17-release/services/appfunctions/ AOSP AppSearch (appsindexer) https://android.googlesource.com/platform/packages/modules/AppSearch/+/refs/heads/android17release/service/java/com/android/server/appsearch/appsindexer/ android/appfunctions (Testing Agent) https://github.com/android/appfunctions Koog https://github.com/JetBrains/koog Issue Tracker https://issuetracker.google.com/issues/490650368 (not called from Gemini) https://issuetracker.google.com/issues/539569496 (multi-module) https://issuetracker.google.com/issues/551999767 (KDoc language) Demo repository 74