Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Chengjia Wang, Guang Yang, Giorgos Papanastasiou, Heye Zhang*, Joel J. P. C. Rodrigues, Victor Hugo C. De Albuquerque

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

20 Citations (Scopus)

Abstract

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different subtasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks into light-weight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, in this article, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world Internet of Things (IoT) scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new 'brain storming' mechanism and optimizable temperature parameters. In our comparison experiments, this HBS method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of authors' knowledge, this is the first IoT-oriented method that allows asynchronous bidirectional heterogeneous knowledge distillation in deep networks.

Original languageEnglish
Pages (from-to)5511-5521
Number of pages11
JournalIEEE Transactions on Industrial Informatics
Volume17
Issue number8
Early online date7 Jul 2021
DOIs
Publication statusPublished - Aug 2021

Keywords

  • Deep learning
  • heterogeneous classifiers
  • Internet of Things (IoT)
  • knowledge distillation (KD)
  • online learning

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Information Systems
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation'. Together they form a unique fingerprint.

Cite this