GPU sharing done right (SCaLE 23x)

Multi-tenant Kubernetes with GPU sharing is a compelling model for AI infrastructure, but it requires careful design to balance performance with security. This session shows how to build a secure and scalable environment where multiple teams can run GPU workloads without compromising isolation or access control. We’ll cover open source options like KAI Scheduler and vCluster and demonstrate how to integrate external tools for secret management and dynamic access policies, all within an architecture that lets teams feel like they have their own cluster—while behind the scenes, resources are efficiently pooled and shared.

Transcript

1. ©2026 HASHICORP GPU sharing done right Staff Solutions Architect HashiCorp Adrian Todorov Principal Developer Advocate, Depot Scott McAllister
2. Principal Developer Advocate Depot @stmcallister Scott McAllister
3. Staff Solutions Architect @ HashiCorp, ex-SRE, ex-developer atodorov.me Adrian Todorov
4. Traditional clusters are like gamers.
5. Traditional clusters don't like to share GPU.
6. AI use cases GPU demand Interest in AI What’s happening
7. AI use cases GPU demand Interest in AI What’s happening
8. Multi-Process Service (MPS) Process-1 Process-2 Process-3 GPU CUDA Context
9. Multi-Process Service (MPS) Process-1 Process-2 Process-3 GPU CUDA Context
10. Multi-Process Service (MPS) Process-1 Process-2 Process-3 GPU CUDA Context
11. GPU User User User User User Multi-Instance GPUs (MIG) GPU instance 0 GPU instance 1 GPU instance 2 GPU instance 3 GPU instance 4
12. GPU User User User User User Multi-Instance GPUs (MIG) GPU instance 0 GPU instance 1 GPU instance 2 GPU instance 3 GPU instance 4
13. GPU User User User User User Multi-Instance GPUs (MIG) GPU instance 0 GPU instance 1 GPU instance 2 GPU instance 3 GPU instance 4
14. GPU User User User User User Multi-Instance GPUs (MIG) GPU instance 0 GPU instance 1 GPU instance 2 GPU instance 3 GPU instance 4
15. Time-slicing (Or: custom scheduling) GPU Compute Process 1 Process 2 Process 3 Process 4 T2 T3 T4 T1 T2 T3 T4 T1 T1 T2 Time slice
16. Time-slicing (Or: custom scheduling) GPU Compute Process 1 Process 2 Process 3 Process 4 T2 T3 T4 T1 T2 T3 T4 T1 T1 T2 Time slice
17. Time-slicing (Or: custom scheduling) GPU Compute Process 1 Process 2 Process 3 Process 4 T2 T3 T4 T1 T2 T3 T4 T1 T1 T2 Time slice
18. Time-slicing (Or: custom scheduling) GPU Compute Process 1 Process 2 Process 3 Process 4 T2 T3 T4 T1 T2 T3 T4 T1 T1 T2 Time slice
19. Third-Party Scheduling Tools
20. Third-Party Scheduling Tools
21. Per-team GPU cluster Tenant namespaces GPU GPU KAI Tenant A Tenant namespaces GPU GPU KAI Tenant B
22. Per-team GPU cluster Tenant namespaces GPU GPU KAI Tenant A Tenant namespaces GPU GPU KAI Tenant B
23. Per-team GPU cluster Tenant namespaces GPU GPU KAI Tenant A Tenant namespaces GPU GPU KAI Tenant B
24. Namespace-based multi-tenancy tenant-ns-a GPU GPU KAI tenant-ns-b GPU GPU Tenant A Tenant B
25. Namespace-based multi-tenancy tenant-ns-a GPU GPU KAI tenant-ns-b GPU GPU Tenant A Tenant B
26. Namespace-based multi-tenancy tenant-ns-a GPU GPU KAI tenant-ns-b GPU GPU Tenant A Tenant B
27. Namespace-based multi-tenancy tenant-ns-a GPU GPU KAI tenant-ns-b GPU GPU Tenant A Tenant B
28. pod-1 custom-resource Team-shared cluster deployment pod-1 custom- Data store Control vcluster syncer API server Namespace in virtual cluster Virtual 
 cluster context etcd Control Scheduler API server Control plane synced-pod-1 vcluster-pod Namespace in K8s K8s 
 cluster context vcluster-pod Tenant Connects to cluster API server Controls virtual cluster Admin Connects to K8s API server Controls K8s context
29. Team-shared cluster etcd Control Scheduler API server Control plane pod-1 Namespace in team clusters Team-cluster context custom-resource
31. GPU sharing only works when you understand your trust boundaries.
32. but not isolation. Namespaces provide 
 logical segmentation,
33. Per-team clusters 
 address isolation, but not efficiency.
34. Multi-team clusters 
 address efficiency, but not isolation.
35. 02 Where we need to be at
36. How do you deal with cluster-level resource?
37. Multi-tenant cluster Tenant A Tenant B tenant-a-proj-1 tenant-b-proj-1 Kubernetes cluster Other Operations vcluster-a vcluster-b DGX node DGX node DGX node DGX node
38. Team-specific cluster etcd Control manager Scheduler API server Control plane pod-1 custom-resource Deployments
39. Team-specific cluster etcd Control Scheduler API server Control plane pod-1 Deployments custom-resource Vault
40. Why Vault Client Vault Authentication AD LDAP IAM
41. How Vault Works Client Vault Authentication AD LDAP IAM Static Key/Value secrets Dynamic Cloud API credentials Dynamic Database Credentials PKI certificates
42. Eliminate forever credentials
43. Audit All The Things
44. Vault and Kubernetes \ Pod Vault Vault Secrets Operator Secret pod-1 pod-2 pod-3
45. Team-specific security etcd Control manager Scheduler Secrets Team 2 Namespaces Team 3 Team 1 Control plane Deployments VSO pod-1 Namespace
46. Team-shared cluster Access for 4h Control plane Access for S3 Team 1 Team 2 Deployments VSO Pod-53 Namespace Control plane Deployments VSO Pod-1 Namespace
47. Configuration
48. --- apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultConnection spec: address: http://vault.vault.svc.cluster.local:8200 --- apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultAuth metadata: namespace: fi netune name: vault-auth spec: vaultConnectionRef: vault-connection method: kubernetes kubernetes: serviceAccount: fi netune role: prod namespace: fi netune Kubernetes Configuration vault.yaml
49. --- apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultDynamicSecret metadata: namespace: fi netune name: vault-dynamic-secret-db-prod spec: vaultAuthRef: vault-auth mount: db path: creds/pgsql-prod destination: create: true name: pgsql-prod Kubernetes Configuration vault.yaml
50. $ vault write auth/kubernetes/role/demo bound_service_account_names=special-job-prod bound_service_account_namespaces= fi netune policies= fi netune-prod,prod,database-prod ttl=24h 
 $ cat database-prod.policy.hcl path "db/creds/pgsql-prod" { capabilities = ["read"] } Vault Configuration vault.yaml
51. 03 What we need to remember
52. Architect for failure.
53. Isolation is key.
54. Defense in-depth is defense that works.
55. Virtual clusters limit blast-radius.
56. Virtual clusters enable easier auditing.
57. Thank you speakerdeck.com/stmcallister atodorov.me