[DevTalksRomania]Building Augmented Reality Experiences with Flutter and AI-Powered Development

This is the slide, "Building Augmented Reality Experiences with Flutter and AI-Powered Development" at Dev Talks Romania
#DevTalks #DevTalksRomania #GenerativeAI #mobile #flutter

Transcript

1. Kenichi Kambara (@korodroid) June 5, 2025 Building Augmented Reality Experiences with Flutter and AI-Powered Development Romania 2025
2. •Mobile App Development •Talks (21 Int’l/100+ Domestic) •Writings (10 Dev/Edu Books) •[Of fi cial] Evangelist at NTT TechnoCross e.g. AI-powered call analysis, livestock monitoring, and privileged access management •[Private] iplatform.org About me Kenichi Kambara (X:@korodroid)
3. Sekaiphone Pro(Flutter(+Kotlin/Swift)) Behind the Scenes of My Personal Project
4. My Product and What I’ve Learned  Visited 50+ Countries
5. Agenda •My Motivation and Vision •Enhancing your Flutter App with AR •Common Design Pitfalls in AR UI/UX to Avoid •AI-Driven AR Development •Best Practices Based on Our Experience
6. My Motivation and Vision  From Smartphones to the Seamless Integration of the Virtual ˍ Physical 🚀
7. Use Case 1: Same Place, No Face-to-Face 
8. Use Case 1: My Solution with AR 
9.  Use Case 2: Understanding Unfamiliar Text
10.  Use Case 2: My Solution with AR
11. Challenges to Overcome  •Steps to Enhance a Flutter App with AR •Common Development Challenges and Solutions •Principles for Developing AR Features
12. About Flutter  A framework that allows simultaneous development of apps for multiple platforms from a common source (Cross-platform) https://docs. fl utter.dev/
13. 3 Unique features of Flutter  1. Beautiful UI & High Performance 2. Fast and ef fi cient development process 3. Uses Dart as the programming language
14. 1. Beautiful UI & High Performance  Flutter showcase https:// fl utter.dev/showcase
15. 2. Fast and ef fi cient development process  Hot reload https://docs. fl utter.dev/tools/hot-reload
16. 3. Uses Dart as the programming language  Very similar to Java/JavaScript import 'package:flutter/material.dart'; void main() { runApp( Center( child: Text( ‘Hello, Flutter’, textDirection: TextDirection.ltr, ), ), ); }
17. Enhancing your Flutter App with AR
18. Use Case 1: Camera Preview & Overlay  (a) Create a Camera Preview Screen (b) Stack multiple Layers
19. (a) Create a Camera Preview Screen  1. Install the package 2. Set up required permissions 3. Initialize the camera & show the preview 4. Test the app
20.  1. Install the package https://pub.dev/packages/camera
21.  1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0
22.  1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0
23.  2. Set up required permissions [Android]AndroidManifest.xml [iOS]ios/Runner/Info.plist
24.  3. Initialize the camera & show the preview main.dart 1/3
25.  3. Initialize the camera & show the preview main.dart 2/3
26.  3. Initialize the camera & show the preview main.dart 3/3
27.  4. Test the app Device Emulator
28. (b) Stack multiple Layers  A B A B
29. How to Stack multiple Layers  ref. Stack: https://api. fl utter.dev/ fl utter/widgets/Stack-class.html
30.  Code Example (Multiple Layers) A B Widget _stackTwoScreen() { return Stack( children: [ ScreenA, ScreenB, ], ); }
31. Use Case 2: Text Recognition and Overlay  (c) Recognize Text [Translate recognized Text] (d) Overlay Text on Camera Preview (e) Adjust Text Position Lapte
32. (c) Recognize Text  1. Install the package 2. Recognize Text from Camera Preview 3. Test the app
33.  1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0 google_mlkit_text_recognition: ^0.10.0
34.  2. Recognize Text from Camera Preview CameraImage RecognizedText InputImage Convert Recognize Text MLKit - TextRecognizer#processImage
35.  2. Recognize Text from Camera Preview main.dart final TextRecognizer _textRecognizer = TextRecognizer(); void _processCameraImage(CameraImage image) async { if (_isDetecting) return; _isDetecting = true; try { final WriteBuffer allBytes = WriteBuffer(); for (var plane in image.planes) { allBytes.putUint8List(plane.bytes); } final bytes = allBytes.done().buffer.asUint8List(); // Create Metadata final InputImageMetadata metadata = InputImageMetadata( size: Size(image.width.toDouble(), image.height.toDouble()), rotation: InputImageRotation.rotation0deg, format: InputImageFormat.nv21, // Image format bytesPerRow: image.planes[0].bytesPerRow, );
36.  2. Recognize Text from Camera Preview final InputImage inputImage = InputImage.fromBytes( bytes: bytes, metadata: metadata, ); final RecognizedText recognizedText = await _textRecognizer.processImage(inputImage); // Output recognized Text & Coordination for (TextBlock block in recognizedText.blocks) { final Rect boundingBox = block.boundingBox; final String text = block.text; // Output to console print('Text: $text'); print('BoundingBox: ${boundingBox.left}, ${boundingBox.top}, ${boundingBox.right}, $ {boundingBox.bottom}'); } } catch (e) { print('Error in recognizing text: $e'); } finally { _isDetecting = false; } }
37.  Device Console Text: Milk BoundingBox: 123.0, 214.0, 552.0, 269.0 3. Test the app (on Device)
38.  Device Emulator ? Challenge: Testing AR Features on an Emulator
39.  1. Use the advanced Emulator Setting 2. Use the static Image instead of Camera Solution: How to test using Emulator
40. Using the advanced Emulator Setting  •None •VirtualScene •Emulated •Webcam0
41. Camera Settings on Emulator  Emulated VirtualScene Webcam
42. Using the static Image instead of Camera  Camera Image (Camera Preview) Static Image (Assets)
43. Tips. How to test using Asset/Images  final inputImage = InputImage.fromFilePath('assets/test_image.png'); final RecognizedText recognizedTextResult = await _textRecognizer.processImage(inputImage); Future<RecognizedText> processImage(InputImage inputImage) async { final result = await _channel.invokeMethod( 'vision#startTextRecognizer', <String, dynamic>{ 'id': id, 'imageData': inputImage.toJson(), 'script': script.index }); return RecognizedText.fromJson(result); }
44. (d) Overlay Text on Camera Preview  Lapte Lapte
45.  Code Example (CameraPreview&Text) Widget _stackTwoScreens() { return Stack( children: [ PreviewScreen, OverlayTextScreen, ], ); } Lapte
46. Common Design Pitfalls in AR UI/UX to Avoid
47.  Common Design Pitfalls in AR UI/UX to Avoid •Common user frustrations →e.g., unreadable text, visual overload, slow response •Accessibility issues in AR →e.g., low contrast, small font sizes ,and so on •What I learned from user feedback in my app →e.g., Couldn’t read the overlay in sunlight
48.  e.g. Text Visibility on Camera Preview Milk Milk Milk
49.  • Add a Semi-Transparent Background to the Text • Add a Shadow to the Text • Use Outlined Text • Dynamically Change the Text Color • Blur Part of the Background Solutions to Improve Text Visibility on Camera Preview
50.  Solution examples Semi-Transparent Background Shadow to the Text Outlined Text
51.  • Add a Semi-Transparent Background to the Text • Add a Shadow to the Text • Use Outlined Text • Dynamically Change the Text Color • Blur Part of the Background Solutions to Avoid Display Issues
52.  Add a Semi-Transparent Background to the Text Widget _showOverlayTextScreen() { return Container( color: Colors.black.withOpacity(0.5), child: Text( text, style: textStyle, ), ); }
53.  TextElement elementNew = TextElement( text: resultText, // e.g. “Milk”, boundingBox: element.boundingBox, cornerPoints: element.cornerPoints, symbols: element.symbols, recognizedLanguages: element.recognizedLanguages, confidence: element.confidence, angle: element.angle); Text: Milk BoundingBox: 123.0, 214.0, 552.0, 269.0 (e) Adjust Text Position
54.  (e) Adjust Text Position Bounding box coordinates are based on the original image dimensions Essential to Scale
55.  scaleX = constraints.maxWidth / imageSize.width; scaleY = constraints.maxHeight / imageSize.height; Positioned( left: element.boundingBox.left * scaleX, top: element.boundingBox.top * scaleY, width: element.boundingBox.width * scaleX, height: element.boundingBox.height * scaleY, child: Text( element.text, …) ); (e) Adjust Text Position
56. AI-Driven AR Development
57. How AI Helped Us Accelerate AR Development  •Real Prompts from Our Work fl ow •AI for bug fi xing and optimization •Development speed boost
58. Mobile Apps Development with Generative AI  Natural language prompts for: •Cultural UX Guidance •AR Story Generation •Natural Language Tech Support $POWFSTBUJPOBM
    "* 🧠Gemini 🧠ChatGPT 🧠Claude *%&*OUFHSBUFE
    "* 🛠Cursor 🛠Cline 🛠GitHub Copilot Embedded in your IDE for: •AR Code Optimization •Localization Bug Detection •AR Asset Integration
59. Use Case 1: Using Conversational AI  English Arabic Japanese
60. Examples of Language Resources  // English Resource "en": { // Top Screen "appName": "SekaiPhone Pro", "modeTalkSpeech": "Talk Mode\n(Speech)", "modeTalkText": "Talk Mode\n(Text)", "modePhone": "Phone Mode", "modeCamera": "Camera Mode", // Hoge Screen // … },
61. Preparations with Gemini 
62. Use Case 2: Using IDE-Integrated AI  Camera Preview "EEJOH
    UIF
    DBNFSB
    QBDLBHF 8SJUJOH
    UIF
    DBNFSB
    MPHJD #VJMEJOH
    UIF
    6*
    GPS
    QSFWJFX
    EJTQMBZ BOE
    NPSF
63. Accelerating with Cursor 
64. Important Issues to Consider  User Permission 6TJOH
    4QFDJBM
    'FBUVSFT
    MJLF
    $BNFSB 4IPXJOH
    B
    1FSNJTTJPO
    %JBMPH )BOEMJOH
    UIF
    3FTVMU 0OF
    PG
    UIF
    LFZ
    EJ ff FSFODFT
65. Accelerating with Cursor 
66. Best Practices Based on Our Experience
67. Best Practices based on my experience •Behaviours depend on each device →Testing on many devices is essential •Many operations related on AR are heavier →Performance Tuning is also crucial •On-Device vs Cloud: Considerations →Depending on Use Case 
68. Reference  X: @korodroid
69. AR Features Can Make Your App More Engaging 🚀 Conclusion
70. Facebook:http://fb.com/kanbara.kenichi X:@korodroid LinkedIn:http://www.linkedin.com/in/korodroid Thank you so much Feel free to reach out for technical speeches, writing, or other inquiries