SmaSH: One-shot model architecture search through hypernetworks

Andrew Brock; Theodore Lim; James Millar Ritchie; Nicholas J. Weston

SmaSH: One-shot model architecture search through hypernetworks

Andrew Brock
, Theodore Lim
, James Millar Ritchie
, Nicholas J. Weston

Research output: Contribution to conference › Paper › peer-review

Abstract

Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model’s architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks.

Original language	English
Publication status	Published - May 2018
Event	6th International Conference on Learning Representations 2018 - Vancouver Convention Center, Vancouver, Canada Duration: 30 Apr 2018 → 3 May 2018 Conference number: 6 https://iclr.cc/

Conference

Conference	6th International Conference on Learning Representations 2018
Abbreviated title	ICLR 2018
Country/Territory	Canada
City	Vancouver
Period	30/04/18 → 3/05/18
Internet address	https://iclr.cc/

Keywords

Deep Learning
Hypernetworks
Transfer leanring
Benchmarking

ASJC Scopus subject areas

Language and Linguistics
Education
Computer Science Applications
Linguistics and Language

Access to Document

https://arxiv.org/pdf/1708.05344.pdf

Cite this

@conference{75ee6aba81434b879bba9a005f62a064,

title = "SmaSH: One-shot model architecture search through hypernetworks",

abstract = "Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model{\textquoteright}s architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks.",

keywords = "Deep Learning, Hypernetworks, Transfer leanring, Benchmarking",

author = "Andrew Brock and Theodore Lim and Ritchie, \{James Millar\} and Weston, \{Nicholas J.\}",

year = "2018",

month = may,

language = "English",

note = "6th International Conference on Learning Representations 2018, ICLR 2018 ; Conference date: 30-04-2018 Through 03-05-2018",

url = "https://iclr.cc/",

}

TY - CONF

T1 - SmaSH

T2 - 6th International Conference on Learning Representations 2018

AU - Brock, Andrew

AU - Lim, Theodore

AU - Ritchie, James Millar

AU - Weston, Nicholas J.

N1 - Conference code: 6

PY - 2018/5

Y1 - 2018/5

N2 - Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model’s architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks.

AB - Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model’s architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks.

KW - Deep Learning

KW - Hypernetworks

KW - Transfer leanring

KW - Benchmarking

UR - https://www.scopus.com/pages/publications/85083953913

M3 - Paper

AN - SCOPUS:85083953913

Y2 - 30 April 2018 through 3 May 2018

ER -

SmaSH: One-shot model architecture search through hypernetworks

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this