Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

Robert James Stewart; Andrew Nowlan; Pascal Bacchus; Quentin Ducasse; Ekaterina Komendantskaya

doi:10.3390/electronics10040396

Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

Robert James Stewart, Andrew Nowlan, Pascal Bacchus, Quentin Ducasse, Ekaterina Komendantskaya

Research output: Contribution to journal › Article › peer-review

17 Citations (Scopus)

131 Downloads (Pure)

Abstract

This paper compares the latency, accuracy, training time and hardware costs of neural networks compressed with our new multi-objective evolutionary algorithm called NEMOKD, and with quantisation. We evaluate NEMOKD on Intel's Movidius Myriad X VPU processor, and quantisation on Xilinx's programmable Z7020 FPGA hardware. Evolving models with NEMOKD increases inference accuracy by up to 82% at the cost of 38% increased latency, with throughput performance of 100–590 image frames-per-second (FPS). Quantisation identifies a sweet spot of 3 bit precision in the trade-off between latency, hardware requirements, training time and accuracy. Parallelising FPGA implementations of 2 and 3 bit quantised neural networks increases throughput from 6k FPS to 373k FPS, a 62x speedup.

Original language	English
Article number	396
Journal	Electronics
Volume	10
Issue number	4
DOIs	https://doi.org/10.3390/electronics10040396
Publication status	Published - 5 Feb 2021

Keywords

Evolutionary algorithm
FPGA
Movidius VPU
Neural network
Quantisation

ASJC Scopus subject areas

Control and Systems Engineering
Signal Processing
Hardware and Architecture
Computer Networks and Communications
Electrical and Electronic Engineering

Access to Document

10.3390/electronics10040396Licence: CC BY

Download pdf
© 2021 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Final published version, 1.32 MBLicence: CC BY

Cite this

@article{20adaa64a8da4bc1af80d9bc674aab2a,

title = "Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm",

abstract = "This paper compares the latency, accuracy, training time and hardware costs of neural networks compressed with our new multi-objective evolutionary algorithm called NEMOKD, and with quantisation. We evaluate NEMOKD on Intel's Movidius Myriad X VPU processor, and quantisation on Xilinx's programmable Z7020 FPGA hardware. Evolving models with NEMOKD increases inference accuracy by up to 82% at the cost of 38% increased latency, with throughput performance of 100–590 image frames-per-second (FPS). Quantisation identifies a sweet spot of 3 bit precision in the trade-off between latency, hardware requirements, training time and accuracy. Parallelising FPGA implementations of 2 and 3 bit quantised neural networks increases throughput from 6k FPS to 373k FPS, a 62x speedup.",

keywords = "Evolutionary algorithm, FPGA, Movidius VPU, Neural network, Quantisation",

author = "Stewart, {Robert James} and Andrew Nowlan and Pascal Bacchus and Quentin Ducasse and Ekaterina Komendantskaya",

note = "Funding Information: Funding: This research was funded by EPSRC project “Border Patrol: Improving Smart Device Security through Type-Aware Systems Design (EP/N028201/1)”; EPSRC project “Serious Coding: A Game Approach To Security For The New Code-Citizens (EP/T017511/1)”; National Cyber Security Center, UK, Grant “SecConn-NN: Neural Networks with Security Contracts—towards lightweight, modular security for neural networks”; UK Research Institute in Verified Trustworthy Software Systems research project “CONVENER: Continuous Verification of Neural Networks” (from the “Digital Security Through Verification” call). Publisher Copyright: {\textcopyright} 2021 by the authors. Licensee MDPI, Basel, Switzerland. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = feb,

day = "5",

doi = "10.3390/electronics10040396",

language = "English",

volume = "10",

journal = "Electronics",

issn = "2079-9292",

publisher = "MDPI",

number = "4",

}

TY - JOUR

T1 - Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

AU - Stewart, Robert James

AU - Nowlan, Andrew

AU - Bacchus, Pascal

AU - Ducasse, Quentin

AU - Komendantskaya, Ekaterina

N1 - Funding Information: Funding: This research was funded by EPSRC project “Border Patrol: Improving Smart Device Security through Type-Aware Systems Design (EP/N028201/1)”; EPSRC project “Serious Coding: A Game Approach To Security For The New Code-Citizens (EP/T017511/1)”; National Cyber Security Center, UK, Grant “SecConn-NN: Neural Networks with Security Contracts—towards lightweight, modular security for neural networks”; UK Research Institute in Verified Trustworthy Software Systems research project “CONVENER: Continuous Verification of Neural Networks” (from the “Digital Security Through Verification” call). Publisher Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/2/5

Y1 - 2021/2/5

N2 - This paper compares the latency, accuracy, training time and hardware costs of neural networks compressed with our new multi-objective evolutionary algorithm called NEMOKD, and with quantisation. We evaluate NEMOKD on Intel's Movidius Myriad X VPU processor, and quantisation on Xilinx's programmable Z7020 FPGA hardware. Evolving models with NEMOKD increases inference accuracy by up to 82% at the cost of 38% increased latency, with throughput performance of 100–590 image frames-per-second (FPS). Quantisation identifies a sweet spot of 3 bit precision in the trade-off between latency, hardware requirements, training time and accuracy. Parallelising FPGA implementations of 2 and 3 bit quantised neural networks increases throughput from 6k FPS to 373k FPS, a 62x speedup.

AB - This paper compares the latency, accuracy, training time and hardware costs of neural networks compressed with our new multi-objective evolutionary algorithm called NEMOKD, and with quantisation. We evaluate NEMOKD on Intel's Movidius Myriad X VPU processor, and quantisation on Xilinx's programmable Z7020 FPGA hardware. Evolving models with NEMOKD increases inference accuracy by up to 82% at the cost of 38% increased latency, with throughput performance of 100–590 image frames-per-second (FPS). Quantisation identifies a sweet spot of 3 bit precision in the trade-off between latency, hardware requirements, training time and accuracy. Parallelising FPGA implementations of 2 and 3 bit quantised neural networks increases throughput from 6k FPS to 373k FPS, a 62x speedup.

KW - Evolutionary algorithm

KW - FPGA

KW - Movidius VPU

KW - Neural network

KW - Quantisation

UR - http://www.scopus.com/inward/record.url?scp=85100456075&partnerID=8YFLogxK

U2 - 10.3390/electronics10040396

DO - 10.3390/electronics10040396

M3 - Article

SN - 2079-9292

VL - 10

JO - Electronics

JF - Electronics

IS - 4

M1 - 396

ER -

Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this