Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Model Mondays S2E08: On-Device and Local AI

Model Mondays S2E08: On-Device and Local AI

On-device inference is critical if you want to run AI models locally, on your own hardware, e.g., for edge computing needs. Join as as we talk to Maanav Dalal about Foundry Local – a solution built on ONNX Runtime (for use in CPUs, NPUs & GPUs) & taking you from prototype to production.

On our customer stories segment we're also joined by Marilyn Morgan Westner (Co-founder and Chief Experience Officer) and Alex Westner, (Co-founder and CEO) at Xander Glasses. Learn how they bring sight to sound with AI-driven captioning on their wearable assistive device.

Register:
Livestreamed: Aug 04 / https://developer.microsoft.com/en-us/reactor/events/26127/
Discord AMA: Aug 08 / https://discord.gg/azureaifoundry?event=1382863345777901670

Catch-up After:
Livestream Recording: https://www.youtube.com/watch?v=ILBDDCJ0d9g
AMA Transcript: https://github.com/orgs/azure-ai-foundry/discussions/108
AMA Schedule: https://aka.ms/model-mondays/forum

About Model Mondays:
GitHub: https://aka.ms/model-mondays/
Livestream: https://aka.ms/model-mondays/rsvp
Discord: https://aka.ms/model-mondays/discord

Avatar for Nitya Narasimhan, PhD

Nitya Narasimhan, PhD

August 04, 2025
Tweet

More Decks by Nitya Narasimhan, PhD

Other Decks in Technology

Transcript

  1. Maanav Dalal Product Manger, Core AI / Microsoft MODEL MONDAYS

    S2:E08 Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:15 PM ET Tools Spotlight On: On-Device & Local AI
  2. Co-founder & Chief Experience Officer, Xander MODEL MONDAYS S2:E08 Marilyn

    Morgan Westner Co-founder & CEO, Xander Alex Westner Customer Stories: Xander Industry Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:15 PM ET
  3. ❶ · Try · RFT Observability RFT Observability Provides real-time,

    in-depth visibility into your RFT job by automatically kicking off evaluation (“auto-evals”) that shows the detailed finetuning progress at each checkpoint.. Read:| https://techcommunity.microsoft.com/category/ai/blog/azure-ai-services-blog
  4. ❷ · Try · Github Spark Github Spark Build and

    ship full-stack intelligent apps using natural language with access to the full power of the GitHub platform—no setup, no configuration, and no headaches. Read: https://github.blog/changelog/2025-07-23-github-spark-in-public-preview-for-copilot-pro-subscribers/
  5. ❸ · Try · DAViD Models DAVID These are models

    that require only a fraction of the cost of training and inference when compared with foundational models of similar accuracy SynthHuman Dataset: https://github.com/microsoft/DAViD Read Paper: https://arxiv.org/abs/2507.15365
  6. ❹ · Try · Agent Experience Optimization Agent Experience Optimization

    Make your content easily discoverable by AI agents recent best practices, including how Microsoft’s NLWeb project fits in, and other emerging standards Explore Series: https://aka.ms/the-future-of-ai View: https://aka.ms/model-mondays/forum
  7. ❺ · Watch · MCP for Beginners MCP For Beginners

    core concepts, hands-on development, advanced implementation, and real-world case studies. Watch: https://aka.ms/mcp-videos View: https://aka.ms/mcp-for-beginners
  8. Maanav Dalal Product Manger, Core AI / Microsoft On-Device &

    Local AI MODEL MONDAYS S2:E08 Hosted By: Nitya Narasimhan, PhD AUG 04, 2025 · 1:30-2:00 PM ET Tools
  9. ❶ · Read · Documentation Foundry Local enables efficient, secure,

    and scalable AI model inference directly on your devices! Understand the Foundry Local Architecture Docs: https://learn.microsoft.com/azure/ai-foundry/foundry-local Join: https://aka.ms/model-mondays/discord Key features include Local inference (on-device) with data privacy, Model customization with cost efficiency (local hardware), Seamless integrations with CLI, SDK and REST API
  10. ❷ · Watch · Build Session Foundry Local – Building

    cutting-edge on-device AI experiences Build on-device AI with Azure AI Foundry, ONNX Runtime & Rich Tooling Watch: https://build.microsoft.com/sessions/BRK146 View: https://aka.ms/model-mondays/forum Build intelligent on-device AI with Azure AI Foundry. With seamless integration and top performance across CPU, GPU, and NPU, as well as the seamless integration of Azure AI Foundry Model catalog and inferencing APIs for great developer experience cross local and cloud. Experience improved performance, enhanced security, and cost-efficient operations that minimize errors while enabling real-time processing both locally and in the cloud.
  11. ❸ · Explore · Repository What link is the QR

    Code pointing to Takeaway message with more context for what you learn from that link Repo: https://github.com/microsoft/Foundry-Local View: https://aka.ms/model-mondays/forum
  12. ❹ · Try · See it in practice Get Hands-on

    with a practical fine-tuning lab using Foundry Local Highlights use of local models & hardware for on-device inference Hands-on Lab: https://github.com/microsoft/Build25-LAB329 View: https://aka.ms/model-mondays/forum
  13. ❺ · Explore · Olive Toolkit Want to optimize AI

    models for targeted hardware? Try the Olive (ONNX Live) toolkit! Helps build optimal ONNX models e.g., for use in Foundry Local Explore: https://github.com/microsoft/Olive View: https://aka.ms/model-mondays/forum
  14. Co-founder and Chief Experience Officer, Xander MODEL MONDAYS S2:E08 Marilyn

    Morgan Westner Co-founder and CEO, Xander Alex Westner Customer Stories: Xander Industry With: Nitya Narasimhan Visit: https://www.xanderglasses.com/
  15. ❶ · Learn · Sight For Sound Xander Captioning Glasses

    let you “see” what people are saying (sight for sound) Glasses listen and transcribe speech into text in real time powered by Azure AI Action: https://aka.ms/model-mondays/customer-stories View: https://www.xanderglasses.com
  16. Monalisa Whalin Principal PM / Core AI @Microsoft Models for

    AI Agents MODEL MONDAYS S2:E09 Models Hosted By: April Gittens AUG 11, 2025 · 1:30-2:00 PM ET
  17. Maanav Dalal Product Manger, Core AI / Microsoft Foundry Friday

    AMA AZURE AI FOUNDRY DISCORD ON-DEVICE AND LOCAL AI AUG 08, 2025 · 1:30-2:00 PM ET Tools
  18. Fri 1:30pm ET Join The Discord https://aka.ms/model-mondays/Discord Links To Resources

    Summary of AMA https://aka.ms/model-mondays/Forum #ModelMondays WATCH LIVE ON MONDAY FIND RECAPS ON FORUM JOIN AMA ON FRIDAY Mon 1:30pm ET Register Here https://aka.ms/model-mondays/RSVP