FreezeOut: Accelerate Training by Progressively Freezing Layers

Andrew Brock; Theodore Lim; James Millar Ritchie; Nicholas J. Weston

FreezeOut: Accelerate Training by Progressively Freezing Layers

Andrew Brock
, Theodore Lim
, James Millar Ritchie
, Nicholas J. Weston

Research output: Contribution to conference › Paper › peer-review

Abstract

The early layers of a deep neural net have the fewest parameters, but take up the
most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.

Original language	English
Number of pages	7
Publication status	Published - 8 Dec 2017
Event	NIPS 2017 Workshop on Optimization: 10th NIPS Workshop on Optimization for Machine Learning - Long Beach, Long Beach, United States Duration: 8 Dec 2017 → … Conference number: 10

Workshop

Workshop	NIPS 2017 Workshop on Optimization
Abbreviated title	NIPS
Country/Territory	United States
City	Long Beach
Period	8/12/17 → …

Keywords

Machine learning
Neural network

ASJC Scopus subject areas

Artificial Intelligence

Access to Document

http://opt-ml.org/papers/OPT2017_paper_32.pdf

Cite this

@conference{5d7f50ae706e467791b0d4287b351a1e,

title = "FreezeOut: Accelerate Training by Progressively Freezing Layers",

abstract = "The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20\% wall-clock time during training with 3\% loss in accuracy for DenseNets on CIFAR.",

keywords = "Machine learning, Neural network",

author = "Andrew Brock and Theodore Lim and Ritchie, \{James Millar\} and Weston, \{Nicholas J.\}",

year = "2017",

month = dec,

day = "8",

language = "English",

note = "NIPS 2017 Workshop on Optimization : 10th NIPS Workshop on Optimization for Machine Learning, NIPS ; Conference date: 08-12-2017",

}

TY - CONF

T1 - FreezeOut: Accelerate Training by Progressively Freezing Layers

AU - Brock, Andrew

AU - Lim, Theodore

AU - Ritchie, James Millar

AU - Weston, Nicholas J.

N1 - Conference code: 10

PY - 2017/12/8

Y1 - 2017/12/8

N2 - The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.

AB - The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. We empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets on CIFAR.

KW - Machine learning

KW - Neural network

UR - https://github.com/ajbrock/FreezeOut

M3 - Paper

T2 - NIPS 2017 Workshop on Optimization

Y2 - 8 December 2017

ER -

FreezeOut: Accelerate Training by Progressively Freezing Layers

Abstract

Workshop

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this