Reconciling Accuracy, Cost, and Latency of Inference Serving Systems

17th Cloud Control Workshop, Sweden, 2024
https://cloudresearch.org/workshops/17th/

University of Florida
Monday, Sept. 16
https://news.ece.ufl.edu/2024/08/14/seminar-pooyan-jamshidi/

Transcript

1. Reconciling Accuracy, Cost, and Latency of Inference Serving Systems Pooyan Jamshidi https://pooyanjamshidi.github.io/ University of South Carolina
3. Multi-Objective Optimization with Known Constraints under Uncertainty Solutions: InfAdapter [2023]: Autoscaling for ML Inference IPA [2024]: Autoscaling for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline Dynamic SLO Problem: Different Assumptions
4. Thank you, Saeid Ghafouri! 4
5. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO
6. “More than 90% of data center compute for ML workload, is used by inference services” 6
7. ML inference services have strict requirements 7 Highly Responsive!
8. ML inference services have strict requirements 8 Highly Responsive! Cost-Efficient!
9. ML inference services have strict requirements 9 Highly Accurate! Highly Responsive! Cost-Efficient!
10. ML inference services have strict & conflicting requirements 10 Highly Accurate! Highly Responsive! Cost-Efficient!
11. More challenge: Dynamic workload 11
12. Resource allocation 12
13. Resource allocation 13
14. Resource allocation 14
15. Resource allocation 15
16. Resource allocation 16
17. Resource allocation 17 Over Provisioning Under Provisioning
18. In ML pipelines, we can now adapt the quality of services, too! 18 ResNet18: Tiger ResNet152: Dog
19. Quality adaptation 19
20. First insight: The same throughput can be achieved with different computing resources by switching the model variants 20
21. Multi-models (our solution—InfAdapter) vs single-model (Model-Switching) Higher average accuracy by using multiple model variants 21
23. InfAdapter: Implementation details 23 Selecting a subset of model variants, each having its size meeting latency requirements for the predicted workload while maximizing accuracy and minimizing resource cost
24. InfAdapter: Formulation 24
25. InfAdapter: Formulation 25 Maximizing Average Accuracy
26. InfAdapter: Formulation 26 Maximizing Average Accuracy Minimizing Resource and Loading Costs
27. InfAdapter: Formulation 27
28. InfAdapter: Formulation 28 Supporting incoming workload
29. InfAdapter: Formulation 29 Supporting incoming workload Guaranteeing end-to-end latency
30. InfAdapter: Design 30
31. InfAdapter: Design 31
32. InfAdapter: Design 32
33. InfAdapter: Experimental evaluation setup Workload: Twitter-trace sample (2022-08) Baselines: Kubernetes VPA and Model-Switching Used models: Resnet18, Resnet34, Resnet50, Resnet101, Resnet152 Interval adaptation: 30 seconds Kubernetes cluster: 48 Cores, 192 GiB RAM 33
34. Workload Pattern 34
35. InfAdapter: P99-Latency evaluation 35
36. InfAdapter: P99-Latency evaluation 36
37. InfAdapter: P99-Latency evaluation 37
38. InfAdapter: P99-Latency evaluation 38
39. InfAdapter: P99-Latency evaluation 39
40. InfAdapter: Accuracy evaluation 40
41. 41 InfAdapter: Cost evaluation
42. InfAdapter: Tradeoff Space 42
43. Takeaway 43 Inference Serving Systems should consider accuracy, latency, and cost at the same time.
44. Takeaway 44 Model variants provide the opportunity to reduce resource costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time.
45. Takeaway 45 Model variants provide the opportunity to reduce resource costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time. InfAdapter!
46. 46 https://github.com/reconfigurable-ml-pipeline/InfAdapter
47. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO
48. Inference Pipeline 48 Video Decoder Stream Muxer Primary Detector Object Tracker Secondary Classifier # Configuration Options 55 86 14 44 86
49. 49 The Variabilities ML Pipelines
50. Search Space
51. Is only scaling enough? ?
52. Effect of Batching
53. 53 Goal: Providing a flexible inference pipeline
54. Problem Formulation Accuracy Objective Resource Objective Batch Control
55. Problem Formulation Latency SLA Throughput Constraint One active model per node
56. Evaluations 56
57. 1. Industry standard 2. Used in recent research 3. Complete set of autoscaling, scheduling, observability tools (e.g. CPU usage) 4. APIs for changing the current AutoScaling algorithms 1. Industry standard ML server 2. Have the ability make inference graph 3. Rest and GRPC endpoints 4. Have many of the features we need like monitoring stack out of the box How to navigate Model Variants
58. 58 Evaluation https://github.com/reconfigurable-ml-pipeline/ipa
59. 59 We compared IPA with RIM and FA2
60. 60 Audio + QA Pipeline
61. 61 Adaptivity to multiple objectives
62. 62 Effect of predictor
63. 63 Gurobi solver scalability
64. Full replication package is available https://github.com/recon fi gurable-ml-pipeline
65. Model Serving Pipeline Is only scaling enough? ? X Snapshot of the System X Adaptivity to multiple objectives
66. InfAdapter [2023]: Autoscaling for ML Model Inference IPA [2024]: Autoscaling for ML Inference Pipeline Sponge [2024]: Autoscaling for ML Inference Pipeline with Dynamic SLO
67. Dynamic User -> Dynamic Network Bandwidths ˻ Users move ˻ Fluctuations in the network bandwidths ˻ Reduced time-budget for processing requests 67 SLO network latency processing latency
68. Dynamic User -> Dynamic Network Bandwidths ˻ Users move ˻ Fluctuations in the network bandwidths ˻ Reduced time-budget for processing requests 68 SLO network latency processing latency
69. Inference Serving Requirements 69 Highly Responsive! Cost-Efficient! Resource Scaling In-place Vertical Scaling Horizontal Scaling (end-to-end latency guarantee) (least resource consumption) (more responsive) (more cost efficient) Sponge!
70. Vertical Scaling DL Model Profiling ˻ How much resource should be allocated to a DL model? ˻ Latency/batch size → linear relationship ˻ Latency/CPU allocation → inverse relationship 70
71. Problem Formulation 71
72. Problem Formulation 72 Minimize resource costs
73. Problem Formulation 73 Limit the batch size to grow infinitely! Minimize resource costs
74. Problem Formulation 74 Limit the batch size to grow infinitely! Minimize resource costs
75. 3 design choices: 1. In-place vertical scaling • Fast response time 2. Request reordering • High priority requests 3. Dynamic batching • Increase system utilization 75 System Design
76. Evaluation SLO guarantees (99th percentile) with up to 20% resource save up compared to static resource allocation. Sponge source code: https://github.com/saeid93/sponge 76
77. Future Directions 77 Resource Scaling In-place Vertical Scaling Horizontal Scaling (more responsive) (more cost efficient) Sponge! How can both scaling mechanisms be used jointly under a dynamic workload to be responsive and cost efficient while guaranteeing SLOs?
78. Performance goals are competing and users have preferences over these goals The variability space (design space) of (composed) systems is exponentially increasing Systems operate in uncertain environments with imperfect and incomplete knowledge Goal: Enabling users to f ind the right quality tradeoff Lander Testbed (NASA) Turtlebot 3 (UofSC) Husky UGV (UofSC) CoBot (CMU)