Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gemini in Android Studio I/O extended 2024

Gemini in Android Studio I/O extended 2024

인천/송도 IO extended 2024에서 위 주제로 발표한 자료 입니다.
https://festa.io/events/5477

Veronikapj

July 27, 2024
Tweet

More Decks by Veronikapj

Other Decks in Programming

Transcript

  1. Gemini API on Android • Gemini Models & AICore •

    Gemini Nano use cases on Android • Integration & SDKs Gemini in Android Studio • Code Suggestions • App Quality Insights • Multimodal inputs (Preview) Demo Gemini in Android Studioܳ ഝਊ೧ ࢲ Android Application ѐߊೞӝ ೠ ׀ী ࠁӝ Agenda
  2. Gemini API on Android Android •Gemini Models & AICore •Gemini

    Nano use cases on Android •Integration & SDKs
  3. Gemini Models On-device ੘সਸ ୊ܻ ೞחؘ ੓যࢲ о੢ ബҗ ੸ੋ

    ݽ؛ ׮নೠ ੘সী Ѧ୛ ࢎਊ ೞӝ જ਷ ୭࢚੄ ݽ؛ Gemini Nano Gemini Pro Ҋب۽ ࠂ੟ೠ ੘সਸ ୊ܻ ೞӝ ਤೠ о੢ ъ۱ೠ ݽ؛ Gemini Ultra
  4. Gemini Models •Android 14+੄ AI Core ۽ ੸ਊ •Google AI

    Edge SDKܳ ా೧ API ੽Ӕ •Pixel 8 Pro৬ Samsung S24 दܻૉীࢲ ࢎਊ оמ •ੋఠ֔ োѾ হ੉ب ੘ز •ѐੋ ੿ࠁ ࠁഐ •ߔূ٘ ੋ೐ۄ ҙܻ ೙ਃࢿ ઁѢ On-device ੘সਸ ୊ܻ ೞחؘ ੓যࢲ о੢ ബҗ ੸ੋ ݽ؛ Gemini Nano
  5. Gemini Nano ׮নೠ హਵ۽ ݫद૑ ਽׹ೞחؘ ب਑ • Magic Rewrite

    • Suggested Text ਺ࢿ ֣਺ ӝ۾੄ झ݃౟ ਃড ࢤࢿ • Highlight main topics of voice transcripts(English only) • Powered via Gemini Nano on eligible devices • Available offline + data processed on-device Messages Recorder ఃࠁ٘ীࢲ ޙ੗ ݫद૑ী ؀ೠ झ݃౟ ਽׹ ઁউ GBoard
  6. AI Core Android on-device AI under the hood, I/O 24

    Client app AI Edge SDK LoRA Gemini Nano Safety features AI Core TPU / NPU / APU High-level inference architecture •AICore ח System Imageী ನೣ •Google Playܳ ా೧ ૑ࣘ੸ਵ۽ সؘ੉౟ •জীࢲח AI Edge SDK۽ ੽Ӕ •Low-Rank Adaptation(LoRA) ۨ੉য۽ জ ѐߊ੗о ݽ؛ ݏ୺ ࢸ੿ оמ https://developer.android.com/ai/aicore
  7. AI Core Android on-device AI under the hood, I/O 24

    Why is Fine-tuning important? •೐܁೐౟ ূ૑פয݂݅ਵ۽ח on-device ݽ؛ীࢲ ୽࠙ೞ૑ ঋ਺ •ઁড ઑѤ : model size, context window •LoRA Fine-tuningਸ ా೧ ੘਷ ݽ؛੉ ౠ੿ ࢎਊ ࢎ۹ীࢲ ڪযդ ࢿמਸ ߊൃೡ ࣻ ੓ب۾ ೣ Client app AI Edge SDK Gemini Nano Safety features AI Core TPU / NPU / APU LoRA https://developer.android.com/ai/aicore
  8. Gemini API Integrate Gemini API Google AI Client SDK Vertex

    AI SDK for Firebase (Preview) (Preview) Add Generative AI to your Android app with the Gemini API, I/O 24
  9. Gemini API Integration 4ױ҅ Add Generative AI to your Android

    app with the Gemini API, I/O 24 Google AI Studioীࢲ ೐܁೐౟ܳ ೐۽షఋ੉ೝ Google AI Studioীࢲ API ః ߊә 01 02 ೐۽ં౟ী Google AI Client SDKী ؀ೠ ઙࣘࢿ ୶о Gemini Proܳ Kotlin ௏٘ী ా೤ 03 04
  10. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey,

    generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
  11. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey,

    generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
  12. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey,

    generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
  13. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey,

    generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
  14. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey,

    generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting( HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE ) ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
  15. scope.launch { val response = model.generateContent( content { image(bitmap) text("What

    is the object in this picture?") } ) } Prompting https://developer.android.com/ai/google-ai-client-sdk
  16. scope.launch { val response = model.generateContentStream( content { image(bitmap) text("What

    is the object in this picture?") } ).collect { response -> outputContent += response.text } } Prompting https://developer.android.com/ai/google-ai-client-sdk
  17. Gemini API Onboarding tool Prototyping SDKs Google AI Studio ীࢲ

    ೐܁೐౟ܳ ࢸ҅೤פ׮. Gemini API ܳ জী ୶о೤פ׮. Google AI Client SDK ܳ ా೤೤פ׮. Vertex AI SDK for Firebase ܳ दب೧ ࠇפ׮.
  18. Code Suggestion ௏٘ ੗ز ৮ࢿ https://developer.android.com/studio/preview/gemini/ai-code-completion Tab ఃܳ ׂ۞ ୶ୌਸ

    ࣻۅೞҊ Escܳ ׂ۞ ୶ୌਸ ૑਒פ׮. ౵ۄ޷ఠ۽ ߉਷ obj ੄ ࣘࢿ ч੉ String ژח Int ੌ ҃਋ ୹۱೧઱Ҋ, Ӓ ৻ীח “Unknown” ਵ۽ ୹۱
  19. Test Scenarios // HomeViewModel.kt /** * Notify that the user

    updated the search query */ fun onSearchInputChanged(searchInput: String) { viewModelState.update { it.copy(searchInput = searchInput) } } ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ਗࠄ ೣࣻ What's new in Android development tools, I/O 24
  20. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟

    // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
  21. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟

    // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
  22. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟

    // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
  23. Resource Translation Resource ߣ৉ೞӝ strings.xml > Gemini > Transform select

    code > “Translate to Korean” Gemini ؀ച ହী ֍যب Resource݅ ߣ৉ػ Ѿҗޛਸ ঳ਸ ࣻ ੓׮.
  24. App Quality Insights Explain this exception from my app running

    on Google sdk_gphone64_arm64 with Android version 15: Exception: <Crash Stack> “Prompt” “Show insights”
  25. Show insights Crash Analysis Report • The Exception • Location

    of the Exception • The Call Stack • How to Fix
  26. @Composable fun Greeting(name: String, modifier: Modifier = Modifier) { Column

    { Text( modifier = modifier .clickable { onTextClicked() }, text = "Click", ) } } fun onTextClicked() { throw RuntimeException("Test exception") } Crash ਬߊ ௏٘
  27. ೧Ѿ ߑߨ • onTextClicked Ѩష: MainActivity.kt ౵ੌ੄ 55ߣ૩ ઴۽ ੉زೞৈ

    onTextClicked ೣࣻ ղ੄ ௏٘ܳ ઱੄ Өѱ Ѩష೤פ׮. RuntimeExceptionਸ ੌਵఆ ࣻ ੓ח ֤ܻо ੓ח૑ ഛੋ ೤פ׮. • ٣ߡӒ: Android Studio੄ ٣ߡѢܳ ࢎਊೞৈ ௏٘ प೯ਸ ױ҅߹۽ ૓೯ೞҊ ৘৻о ߊࢤೞ ח ੿ഛೠ ૑੼ਸ ध߹೤פ׮. ߸ࣻ чҗ प೯ ൒ܴਸ Ѩࢎೞৈ ࢚ടਸ ੉೧೤פ׮. • Ҋ੄੸ੋ ৘৻ ഛੋ: పझ౟ ݾ੸ਵ۽ ৘৻ܳ Ҋ੄۽ ߊࢤदఅ ҃਋, ೧׼ ௏٘ܳ ઁѢೞѢա ઑ ੿೤פ׮. • ৘৻ ୊ܻ: ৘৻о Ҋ੄੸ੋ Ѫ੉ ইצ ҃਋, try-catch ࠶۾җ э਷ ৘৻ ୊ܻܳ ୶оೞৈ ࢚ട ਸ ਋ইೞѱ ୊ܻೞҊ জ੉ ୽جೞ૑ ঋب۾ ೤פ׮. 😎
  28. Coming soon Function Name ୶ୌ Shift + F6 Rename What's

    new in Android development tools, I/O 24 Gemini > Rethink variable names
  29. Demo Layout ࢸݺ [Layout] - A Material 3 `TopAppBar` (all

    padding 10.dp) containing an `OutlinedTextField` with a search icon. - Below, a `LazyColumn` (padding 10.dp) displaying clickable hardcoded search recommendations from 1 to 10.
  30. Demo ӝמ ࢸݺ [Functionality] - The search icon should trigger

    `submitSearch(query: String)` in `SearchScreen` using the text from the `OutlinedTextField`. - Clicking a recommendation should also trigger `submitSearch` in `SearchScreen`.
  31. Demo ઁೠ ࢎ೦ ࢸݺ Include all necessary impo rt s

    with preview composable. Don’t omit code, provide the full solution. Do not include any previous response as context.
  32. Demo Prompt Generate a Jetpack Compose `SearchScreen` composable in package

    `com.example.jetnews.ui.search`. [Layout] - A Material 3 `TopAppBar` (all padding 10.dp) containing an `OutlinedTextField` with a search icon. - Below, a `LazyColumn` (padding 10.dp) displaying clickable hardcoded search recommendations from 1 to 10. [Functionality] - The search icon should trigger `submitSearch(query: String)` in `SearchScreen` using the text from the `OutlinedTextField`. - Clicking a recommendation should also trigger `submitSearch` in `SearchScreen`. Include all necessary impo rt s with preview composable. Don’t omit code, provide the full solution. Do not include any previous response as context.