Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DARTS: Differentiable Architecture Search

Udon
August 05, 2020

DARTS: Differentiable Architecture Search

Hanxiao Liu, Karen Simonyan, Yiming Yang, DARTS: Differentiable Architecture Search, ICLR19.

Udon

August 05, 2020
Tweet

More Decks by Udon

Other Decks in Research

Transcript

  1. Introduction of Neural Architecture Search(1/2) What is Neural Architecture Search?

    • Neural networks and hyper-parameters are still hard to design • Related works • Hyper-parameter optimization[Saxena+, NIPS16], [Bergstra+, JMLR12] • Not architecture-level • Evolution algorithms[Stanley+, Artificial Life 09], [Floreano+, EI08] • Less practical at a large scale Need to construct general framework automatically! 
  2. Introduction of Neural Architecture Search(2/2) Neural Architecture Search with Reinforcement

    Learning[Zoph+ ICLR17] • Proposed reinforcement learning based method with a RNN controller • Not differentiable! RNN controller Heavy heavy computational costs! 
  3. a set of operations, : a weight of a node

    between i and j, apply operation O : α(i,j) o o(x) : • Objectives Cannot compute… Proposed method(1/) Define Differentiable NAS
  4.  where Hessian matrix… So computation is heavy If hyper-parameter

    = 0, then no need to compute! (Discuss later about a classification accuracy) Proposed method(2/) Define Differentiable NAS
  5.  where Approximate a finite difference O(αw) O(α + w)

    Proposed method(3/) Define Differentiable NAS
  6. Conclusion • Achieve differentiable NAS. • Achieve the fastest time

    for training and searching. • Available my implementation from https://github.com/UdonDa/DARTS_pytorch • (Only CIFAR-10)