Putting The Genie in the Bottle - A Crash Course on running LLMs on Android

Think running LLMs requires massive data centers? Think again. A new generation of efficient models is making on-device AI a reality, even on Android.

This session cuts through the hype to explore the practicalities of "Edge Inference." Learn why you'd ditch the cloud API calls for on-device processing.

We'll compare Google's AI Edge SDK and the MediaPipe LLM Inference API, highlighting the trade-offs Android developers face today. Understand the current limitations, the surprising capabilities of modern small models, and why experimenting with on-device LLMs now could give your app a competitive edge tomorrow.

Transcript

1. @iurysza iurysouza.dev Putting The Genie in the Bottle A Crash Course on running LLMs on Android
2. About me •GDE for Android •Senior Engineer @ Klarna @iurysza iurysouza.dev
3. But why tho? Do LLMs on mobile make any sense?
4. •Weren't they super hardware intensive? •We already have Chat-GPT/ Gemini at home on the cloud. But why tho?
5. •SOTA models get the most attention •Benchmarks are crushed every week What gets buried in the news
6. Gemini 2.5 Pro GPT-5 Claude Opus 4.1 Grok 4 Deepseek V3 Mistral Small 3.1 Lamma 3.1 Gemma 3 SmolLM Phi-3 The Model Iceberg
7. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models
8. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models Gemma 3 Model Performance vs. Size
9. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models Mistral 3.1 Model Performance vs. Latency
10. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models Top Left Corner Models
11. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models Model Size vs Eval
12. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models Gemma3 27b ranking in LLM arena Circa March-25
13. •There's another category of breakthroughs happening in parallel •Highly efficient and capable Open Source Models •These models are hitting benchmarks which were SOTA 6 months ago. Top Left Corner Models # of GPUs required vs Elo Score
14. Why Run Locally? Even if they’re getting good, why should you still bother? •Performance & Latency •Privacy & Security •Cost & Accessibility •Offline functionality
15. Two options for running on-device Gen-AI Running on the Edge
16. A system wide LLM •Built in directly in the platform •Gemini Nano • via AI Edge SDK (Oct-24) • via ML Kit GenAI APIs (May-25)
17. A system wide LLM •Built in directly in the platform •Gemini Nano • via AI Edge SDK (Oct-24) • via ML Kit GenAI APIs (May-25)
18. AI Edge SDK •Easiest approach for developers •The model is shared by different apps •Support for LoRA Adapters •Model updates are managed by the platform •Built-in safety features •Available on High-end selected devices only
19. AI Edge SDK •Requirements
20. AI Edge SDK •Requirements • A compatible device •Google: Pixel 9 & 10 series •Samsung: Galaxy S25 series •OnePlus: 13 series •Top Models from: • Honor, Realme, Vivo, Xiaomi • iQOO, POCCO, OPPO, Motorola
21. AI Edge SDK •Requirements • A compatible device • AI Core APK Android AICore App
22. AI Edge SDK •Requirements • A compatible device • AI Core APK • Private Compute Services APK Private Compute Services App
23. AI Edge SDK Using it on your app implementation("com.google.ai.edge.aicore:aicore:0.0.1-exp01")
24. private val model = GenerativeModel( generationConfig = generationConfig { context = applicationContext maxOutputTokens = 600 temperature = 0.9f topK = 16 topP = 0.1f } ) val response = model.generateContentStream(prompt) AI Edge SDK Using it on your app
25. private val model = GenerativeModel( generationConfig = generationConfig { context = applicationContext maxOutputTokens = 600 temperature = 0.9f topK = 16 topP = 0.1f } ) val response = model.generateContentStream(prompt) AI Edge SDK Using it on your app
26. private val model = GenerativeModel( generationConfig = generationConfig { context = applicationContext maxOutputTokens = 600 temperature = 0.9f topK = 16 topP = 0.1f } ) val response = model.generateContentStream(prompt) AI Edge SDK Using it on your app
27. AI Edge SDK Using it on your app With it you can: •Check if the device is supported. •Tune safety settings. •Run inference at high performance. •Optionally, provide a LoRA adapter.
28. AI Edge SDK How does it work? Roughly AI Edge SDK Architecture
29. AI Edge SDK How does it work? AICore •OS-level integration •Model deployment management •Hardware abstraction layer AI Edge SDK Architecture
30. AI Edge SDK How does it work? Private Compute Services/Core •PCS + PCC • Privacy through Data Isolation • PCC: Process data safely. • PCS: Model updates. AI Edge SDK Architecture
31. AI Edge SDK How does it work? Private Compute Services/Core •PCS + PCC • Privacy through Data Isolation • PCC: Process data safely. • PCS: Model updates. Now Playing uses PCC
32. AI Edge SDK How does it work? Private Compute Services/Core •PCS + PCC • Privacy through Data Isolation • PCC: Process data safely. • PCS: Model updates. Smart Reply uses PCC
33. A system wide LLM •Built in directly in the platform •Gemini Nano • via AI Edge SDK (Oct-24) • via ML Kit GenAI APIs (May-25)
34. ML Kit GenAI APIs •High-level APIS for: • Summarization • Proofreading • Rewriting • Image Description
35. Adding it to your App ML Kit GenAI APIs val articleToSummarize = “We are excited to announce a set of on-device ... " val options = SummarizerOptions.builder(context) .setInputType(InputType.ARTICLE) .setOutputType(OutputType.ONE_BULLET) .setLanguage(Language.ENGLISH) .build() val summarizer = Summarization.getClient(options)
36. ML Kit GenAI APIs val articleToSummarize = “We are excited to announce a set of on-device ... " val options = SummarizerOptions.builder(context) .setInputType(InputType.ARTICLE) .setOutputType(OutputType.ONE_BULLET) .setLanguage(Language.ENGLISH) .build() val summarizer = Summarization.getClient(options) Adding it to your App
37. Adding it to your App ML Kit GenAI APIs val articleToSummarize = “We are excited to announce a set of on-device ... " val options = SummarizerOptions.builder(context) .setInputType(InputType.ARTICLE) .setOutputType(OutputType.ONE_BULLET) .setLanguage(Language.ENGLISH) .build() val summarizer = Summarization.getClient(options)
38. ML Kit GenAI APIs val articleToSummarize = “We are excited to announce a set of on-device ... " val options = SummarizerOptions.builder(context) .setInputType(InputType.ARTICLE) .setOutputType(OutputType.ONE_BULLET) .setLanguage(Language.ENGLISH) .build() val summarizer = Summarization.getClient(options) Adding it to your App
39. Two options for running on-device Gen-AI Running on the Edge
40. A system wide LLM •Built in directly in the platform •Gemini Nano • via AI Edge SDK (Oct-24) • via ML Kit GenAI APIs (May-25)
41. •You’re in control • Model management • Setup, initialization, and resource usage •You can run many different models •Bring your own fine tuned model •Wider device compatibility The DIY alternative Self managed Models
42. Adding to your App Media Pipe LLM Inference API What model do we use?
43. Wait, what? 🤨 Adding to your App Media Pipe LLM Inference API
44. What are these models? What's under the hood kaggle.com/models/google/gemma-3
45. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood
46. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood
47. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood Model family ^
48. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood ~1 billion parameters ^
49. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood fine-tuned to follow instructions/commands ^
50. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood weights are quantized to 4-bit integers ^
51. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood weights are quantized to 4-bit integers ^
52. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood MediaPipe bundle format ^
53. ❯ unzip -l gemma3-1b-it-int4.task Archive: gemma3-1b-it-int4.task Length Date Time Name 549971728 03-07-2025 07 : 36 TF_LITE_PREFILL_DECODE 4689074 03-07-2025 07 : 36 TOKENIZER_MODEL 90 03-07-2025 07 : 36 METADATA --------- ------- 554660892 3 files What are these models? What's under the hood
54. TOKENIZER_MODEL •Convert text to and from token IDs •Contains the Vocabulary What are these models? What's under the hood
55. TOKENIZER_MODEL •Convert text to and from token IDs •Contains the Vocabulary { "explain": 123, "this": 456, "model": 789 } What are these models? What's under the hood
56. TF_LITE_PREFILL_DECODE •Model weights & Architecture •Tensor Flow Lite •Prefill & Decode operations What are these models? What's under the hood
57. TF_LITE_PREFILL_DECODE •Model weights & Architecture •Tensor Flow Lite •Prefill & Decode operations What are these models? What's under the hood
58. What are they doing? What's under the hood What do we do with that?
59. We’re running Inference What are they doing? What's under the hood
60. What are they doing? What's under the hood Making next token predictions Running Inference?
61. https: // colab.research.google.com/github/google-ai-edge/ mediapipe-samples/blob/main/codelabs/litert_inference/ Gemma3_1b_fine_tune.ipynb#scrollTo=AM6rDABTXt2F What are they doing? What's under the hood
62. What are they doing? What's under the hood
63. What are they doing? What's under the hood
64. What are they doing? Media Pipe LLM Inference API
65. How to use this in my app? Media Pipe LLM Inference API MediaPipe LLM Inference Thin Java Layer
66. How to use this in my app? Media Pipe LLM Inference API MediaPipe LLM Inference CPP implementation uses LiteRT APIs
67. How to use this in my app? Media Pipe LLM Inference API MediaPipe LLM Inference CPP implementation uses LiteRT APIs
68. Media Pipe LLM Inference API Where were we again?
69. Adding it to your App Media Pipe LLM Inference API What model do we use?
70. Adding it to your App Media Pipe LLM Inference API
71. Adding it to your App Media Pipe LLM Inference API
72. Adding it to your App Media Pipe LLM Inference API What model do we use?
73. Adding it to your App Media Pipe LLM Inference API implementation("com.google.mediapipe:tasks-genai:0.10.24")
74. Adding to your App Media Pipe LLM Inference API # Push the model to the device MODEL_FILE="gemma3-1b-it-int4.task" TARGET_DIR="/storage/emulated/0/Android/data/com.myapp/files/" adb push "$MODEL_FILE" "$TARGET_DIR"
75. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { }
76. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = File( context.getExternalFilesDir(null), modelName ) / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) }
77. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = File( context.getExternalFilesDir(null), modelName ) / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) }
78. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = File( context.getExternalFilesDir(null), modelName ) / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) / / Create session val session = LlmInferenceSession.createFromOptions( llmInference, LlmInferenceSession.LlmInferenceSessionOptions.builder() .setTopK(50) .setTemperature(0.8f) .build() ) }
79. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = File( context.getExternalFilesDir(null), modelName ) / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) / / Create session val session = LlmInferenceSession.createFromOptions( llmInference, LlmInferenceSession.LlmInferenceSessionOptions.builder() .setTopK(50) .setTemperature(0.8f) .build() ) }
80. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = { ... } / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) / / Create session val session = LlmInferenceSession.createFromOptions( llmInference, LlmInferenceSession.LlmInferenceSessionOptions.builder() .setTopK(50) .setTemperature(0.8f) .build() ) / / Send prompt session.addQueryChunk(prompt) session.generateResponseAsync { partialResult, done -> // Handle output Log.d(partialResult) if (done) { // Cleanup llmInference.close() } } }
81. Adding to your App Media Pipe LLM Inference API fun mediaPipeBasicExample( context: Context, modelName: String, prompt: String ) { val modelPath = { .. } / / Create LLM engine val llmInference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .build() ) / / Create session val session = LlmInferenceSession.createFromOptions( llmInference, LlmInferenceSession.LlmInferenceSessionOptions.builder() .setTopK(50) .setTemperature(0.8f) .build() ) / / Send prompt session.addQueryChunk(prompt) session.generateResponseAsync { partialResult, done -> // Handle output Log.d(partialResult) if (done) { // Cleanup llmInference.close() } } }
82. What to use? Bottom Line It depends 😌
83. What to use? Bottom Line Gemini Nano = macOS MediaPipe = Arch Linux
84. What to use? Bottom Line Gemini Nano •Via AIEdge & ML Kit GenAI •Limited to one model •Customization via LoRA Adapters •Guard Rails •Limited availability •OEM adoption is key for its future
85. What to use? Bottom Line MediaPipe •Completely self managed •Niche specialized models •Wider availability
86. CAVEATS Size still matters •Don’t expect SOTA performance •Focus on a narrow use case •Use LoRA Adapters or fine-tuned models. •Handling resource usage is challenging
87. Press X to Doubt "This is a fad.”
88. It's a movie, not a picture “Keep an on the trend lines” 3 ideas •Race to the bottom •History •Specialization + ubiquity
89. Race to the bottom Cloud Models are just too expensive
90. It's a movie, not a picture 1970s-1980s mainframe computer room Remember Mainframes?
91. It's a movie, not a picture Computer terminal Remember Mainframes?
92. It's a movie, not a picture Early personal computer Remember Mainframes?
93. SMOL & Agentic AI Specialization & Ubiquity Mini models everywhere
94. “For many agentic workloads, Small Language Models (SLMs) are a superior default to Large Language Models (LLMs) due to cost, latency, controllability, and fine-tuning ease.” "10–30× cheaper, lower energy, faster responses, easier local deployment” Specialization & Ubiquity Mini models everywhere
95. Specialization & Ubiquity Mini models everywhere Cost reduction enables different business models •Free apps •Premium app tiers •Single payment apps •No variable costs per user interaction with LLMs
96. GBoard Writing Tools Specialization & Ubiquity Now: LoRA adapters everywhere
97. Specialization & Ubiquity Now: LoRA adapters everywhere Pixel Screenshots App
98. Wrapping up Give it a go! Projects for you to get started:
99. Wrapping up Give it a go! AI Edge Gallery
100. Wrapping up Give it a go! Projects for you to get started:
101. Wrapping up Give it a go! flutter_gemma Sample App
102. @iurysza iurysouza.dev Putting The Genie in the Bottle A Crash Course on running LLMs on Android • Google AI Edge SDK Documentation • MediaPipe LLM Inference API • AI on Android Spotlight Week • Paper: Small Language Models are the Future of Agentic AI • Deep Dive into LLMs like ChatGPT • Google AI Edge
103. Thank you! 🇵🇹 @iurysza iurysouza.dev
104. Questions? @iurysza iurysouza.dev