Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FlexiBO: A Decoupled Cost-Aware Multi-Objective...

FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks

AAAI 2024

Pooyan Jamshidi

February 29, 2024
Tweet

More Decks by Pooyan Jamshidi

Other Decks in Science

Transcript

  1. FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks

    Shahriar Iqbal, Jianhai Su, Lars Kotthoff, Pooyan Jamshidi [email protected] AAAI, 24 February 2024 1
  2. One Size Does Not Fit All 1 1.5 2 2.5

    3 3.5 ·104 15 20 25 30 35 40 Energy Consumption (mJ) Prediction Error (%) Xception ← Energy consumption varies 4 × → ← Prediction Error varies 3 × → 2
  3. Heterogeneous Parameters Num of Filters, Filter Size, Learning Rate, Num

    of Epochs DN N Design Compiler Hardware Deployment Num of Active CPUs, CPU/ GPU/ EMC Frequency Cloud, IoT, Edge Num of Threads, GPU Threads, Memory Growth 3
  4. Cost-Unaware Methods Waste Resources Coupled Unaware Pareto Optimal Prediction Error

    (%) Log Wall Clock Time Energy Consumption (mJ) 3000 6000 9000 12000 15 25 35 45 3.65 3.50 3.35 Decoupled Aware Pareto Optimal Prediction Error (%) Log Wall Clock Time Energy Consumption (mJ) 3000 6000 9000 12000 15 25 35 45 3.65 3.50 3.35 4
  5. Proposed Method ▷ weight expected benefit of evaluation by cost

    ▷ choose which objective(s) to evaluate ▷ more efficient use of resources – lower cost, more evaluations 5
  6. Results – Computer Vision 0 50 100 150 200 Cumulative

    Log WallClock Time 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error Xception PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 10000 15000 20000 25000 Energy Consumption (mJ) 15 20 25 30 35 40 Prediction Error (%) Xception PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 6
  7. Results – NLP 0 50 100 150 200 Cumulative Log

    WallClock Time 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error BERT-SQuAD PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 70000 80000 90000 Energy Consumption (mJ) 20 25 30 35 Prediction Error (%) BERT-SQuAD PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 7
  8. Results – Speech Recognition 0 50 100 150 200 250

    300 Cumulative Log WallClock Time 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 Energy Consumption (mJ) 17.5 20.0 22.5 25.0 27.5 30.0 32.5 35.0 Prediction Error (%) DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 8
  9. Results – Evaluations 0 20 40 60 80 100 120

    140 160 180 200 PAL 0 20 40 60 80 100 120 140 160 180 200 PESMO-DEC 2 4 6 8 0 20 40 60 80 100 120 140 160 180 200 Iteration CA-MOBO 0 20 40 60 80 100 120 140 160 180 200 Iteration FlexiBO 2 4 6 8 9
  10. FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization of Deep Neural Networks

    ▷ cost-aware acquisition function decreases cost and improves results ▷ code available at https://github.com/softsys4ai/FlexiBO 0 50 100 150 200 250 300 Cumulative Log WallClock Time 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Hypervolume Error DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 20000 30000 40000 50000 60000 Energy Consumption (mJ) 17.5 20.0 22.5 25.0 27.5 30.0 32.5 35.0 Prediction Error (%) DeepSpeech PAL PESMO ParEGO SMSEGO CA-MOBO PESMO-DEC FLEXIBO-GPLC 10