Neural Architecture Search and Hardware Accelerator Co-Search: A Survey

• https://ieeexplore.ieee.org/document/9606893

Artificial neural networks, Accelerator architectures, Design optimization,
Optimization methods, Machine learning, Image classification, Computer aided
engineering, Approximation methods, Evolutionary computation, Digital circuits

Deep neural networks (DNN) are now dominating in the most challenging
applications of machine learning. As DNNs can have complex architectures with
millions of trainable parameters (the so-called weights), their design and
training are difficult even for highly qualified experts.  In order to reduce
human effort, neural architecture search (NAS) methods have been developed to
automate the entire design process. The NAS methods typically combine searching
in the space of candidate architectures and optimizing (learning) the weights
using a gradient method. In this paper, we survey the key elements of NAS methods
that -- to various extents -- consider hardware implementation of the resulting
DNNs. We classified these methods into three major classes: single-objective NAS
(no hardware is considered), hardware-aware NAS (DNN is optimized for
a particular hardware platform), and NAS with hardware co-optimization (hardware
is directly co-optimized with DNN as a part of NAS). Compared to previous
surveys, we emphasize the multi-objective design approach that must be adopted in
NAS and focus on co-design algorithms developed for concurrent optimization of
DNN architectures and hardware platforms. As most research in this area deals
with NAS for image classification using convolutional neural networks, we follow
this trajectory in our paper. After reading the paper, the reader should
understand why and how NAS and hardware co-optimization are currently used to
build cutting-edge implementations of DNNs.