Share
Embed
How WebAssembly Enhances Edge AI Operations
KubeCon + CloudNativeCon Japan, Japan Community Day
Saza
July 28, 2026
Transcript
39 slides
-
How WebAssembly Enhances Edge AI Operations Japan Community Day July
28, 2026 Soichiro Ueda, Ai Nozaki -
Self Introduction Soichiro Ueda @saza-ku @saza_ku • A Newbie Infrastructure
Engineer • Interest: Cloud Computing, Systems Software Ai Nozaki @ainozaki @ainno321 • Ph.D student at the University of Tokyo • Interest: Hardware Acceleration, Systems Software 2 -
Rise of Edge AI • Edge AI runs inference directly
on edge devices ◦ Enables low-latency local decisions ◦ Protects privacy-sensitive data Measuring advertising effectiveness at 7-Eleven [1] Biometric monitoring by SlateSafety [2] [1]https://www.sony-semicon.com/en/news/2024/2024042401.html [2]https://www.edgeimpulse.com/case-studies/slate-safety-predicting-heat-exhaustion -
Edge AI Requires Updates after Deployment • Edge AI follows
a continuous lifecycle ◦ Update to newer models ◦ Update pre/postprocessing logic Support updating model via over-the-air (OTA) Collect Data Deploy Improve Model -
Problem of Edge AI operation • Device-specific artifacts Python package
container NVIDIA Jetson OS: Ubuntu CPU: Arm Cortex-A -
Problem of Edge AI operation • Device-specific artifacts Python package
container STM specific firmware NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M -
Problem of Edge AI operation • Device-specific artifacts Python package
container STM specific firmware NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M ESP specific firmware ESP OS: FreeRTOS CPU: Xtensa -
The Problem Grows with Scale • Device-specific operation does not
scale ◦ More device families, more build and deploy pipelines ◦ More model and application versions, more artifacts New model New model (C++) Device specific⚠ App Device specific⚠ firm ware -
Toward Cloud-like Operation for Edge AI • Containers provide a
common deployment unit ◦ Applications share a common packaging model ◦ Orchestrators manage life cycles of applications Can we bring this operational model to edge devices? -
Why existing cloud techniques don’t fit? • Resource constrained devices
cannot run containers Feasible but costs much No Linux No Linux ex. nvidia/l4t-pytorch: 5.6GB NVIDIA Jetson STM OS: Ubuntu OS: FreeRTOS/Zephyr CPU: Arm Cortex-A CPU: Arm Cortex-M ESP OS: FreeRTOS CPU: Xtensa -
Why existing cloud techniques don’t fit? • Container images remain
architecture-dependent NVIDIA Jetson OS: Ubuntu CPU: Arm Cortex-A Dev env OS: Ubuntu CPU: x86-64 -
Existing Approaches for the Edge • K3s ◦ Lightweight Kubernetes
distribution ◦ Still requires Linux • KubeEdge ◦ Extends Kubernetes management to edge nodes ◦ Edge devices don’t typically execute workloads ▪ KubeEdge manages them as device resources -
Our Focus: WebAssembly • Wasm is a portable binary format
◦ Compiled from multiple programming languages ◦ Portable across CPUs and OSs ◦ Sandboxed for isolated execution -
WASI: System Interfaces for Wasm • WASI provides standardized interfaces
for Wasm app ◦ ex. Files and streams ◦ ex. Networking Wasm App. Wasm App. WASI Wasm runtime Host kernel -
WASI-NN • Standardized WASI for Neural Networks ◦ Wasm application
invokes inference through WASI-NN ◦ Inference engines remain inside the runtime Wasm App. Wasm App. WASI-NN Wasm runtime Inference Engine • • • • load() set_input() compute() get_output() -
WebAssembly for Edge Devices • Wasm provides a portable and
smaller deployment unit ◦ Does not require a full Linux userspace ◦ Portable across CPU architectures 2.47GB 100MB → Wasm is a promising deployment format for edge devices -
How to manage IoT Devices with Kubernetes Container Runtime Kubelet
Linux Regular Kubernetes Node -
How to manage IoT Devices with Kubernetes Only runs on
Linux Container Runtime Wasm Runtime Kubelet ? Linux Embedded OS Doesn’t run on small devices Regular Kubernetes Node IoT Device -
How to manage IoT Devices with Kubernetes 💡 Edge server
runs Kubelet instead Proxy Kubelet Wasm Runtime Linux Embedded OS Edge Server IoT Device -
Wasm Runtime: waiot • Wasm Runtime targeting small IoT devices
(around 500KiB RAM) • Use WebAssembly Micro Runtime (WAMR) ◦ Manage the runtime ◦ Works like containerd / CRI-O ▪ containerd ↔ waiot ▪ runc ↔ WAMR bytecodealliance/wasm-micro-runtime WAMR waiot Embedded OS • Supported Platforms: ◦ ESP32 ◦ STM32 (WIP) github.com/mewz-project/waiot -
Proxy Kubelet: waiot-kubelet • Works as Kubelets for IoT devices
◦ Runs on Linux Edge Servers ◦ Cooperates with waiot waiot-kubelet • Watch Pod Resources → Make waiot run Wasm ◦ Call remote waiot, instead of local container runtime mewz-project/waiot-kubelet waiot waiot waiot -
How Proxy Kubelet Works control plane created → scheduled to
device-1 waiot-kubelet waiot waiot device-1 device-2 -
How Proxy Kubelet Works control plane device-1 pull waiot-kubelet waiot
waiot device-1 device-2 -
How Proxy Kubelet Works control plane device-1 waiot-kubelet send waiot
waiot device-1 device-2 -
How Proxy Kubelet Works control plane device-1 waiot-kubelet waiot run
device-1 waiot device-2 -
Virtual Kubelet • Library to create your own Kubelet implementation
• Defines Interfaces that must be implemented for Kubelet ◦ e.g. CreatePod, UpdatePod, DeletePod, etc virtual-kubelet/virtual-kubelet -
Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm
runtime Inference Engine -
Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm
runtime ARM TrustZone (TEE) Secure World Inference Engine -
Future Work: NPU/GPU Support Wasm App. Wasm App. WASI-NN Wasm
runtime Secure World Inference Engine -
Conclusion • How to manage Edge AI models? ◦ Frequent
Updates ◦ Constrained Resources → Wasm meets the Edge AI requirements • How to manage Wasm on IoT devices? → Make devices Kubernetes nodes ◦ waiot: Wasm Runtime ◦ waiot-kubelet: Proxy Kubelet • Future Works: AI Inference with GPU/NPU, TrustZone