A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abhijeet Bishnu; Ankit Gupta; Mandar Gogate; Kia Dashtipour; Ahsan Adeel; Amir Hussain; Mathini Sellathurai; Tharmalingam Ratnarajah

doi:10.1109/HealthCom54947.2022.9982772

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abhijeet Bishnu^*, Ankit Gupta, Mandar Gogate, Kia Dashtipour, Ahsan Adeel, Amir Hussain, Mathini Sellathurai, Tharmalingam Ratnarajah^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Citations (Scopus)

Abstract

In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.

Original language	English
Title of host publication	2022 IEEE International Conference on E-Health Networking, Application and Services
Publisher	IEEE
Pages	75-80
Number of pages	6
ISBN (Electronic)	9781665480161
DOIs	https://doi.org/10.1109/HealthCom54947.2022.9982772
Publication status	Published - 21 Dec 2022
Event	2022 IEEE International Conference on E-health Networking, Application and Services - Genoa, Italy Duration: 17 Oct 2022 → 19 Oct 2022

Conference

Conference	2022 IEEE International Conference on E-health Networking, Application and Services
Abbreviated title	HealthCom 2022
Country/Territory	Italy
City	Genoa
Period	17/10/22 → 19/10/22

Keywords

Audio-Visual Speech Enhancement
Downlink
Frame structure
Hearing Technology
Physical layer
Uplink

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Health Informatics

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/HealthCom54947.2022.9982772

Cite this

Bishnu, A., Gupta, A., Gogate, M., Dashtipour, K., Adeel, A., Hussain, A., Sellathurai, M., & Ratnarajah, T. (2022). A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids. In 2022 IEEE International Conference on E-Health Networking, Application and Services (pp. 75-80). IEEE. https://doi.org/10.1109/HealthCom54947.2022.9982772

@inproceedings{696c8de6d5224f9ca0f835196dbe603f,

title = "A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids",

abstract = "In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.",

keywords = "Audio-Visual Speech Enhancement, Downlink, Frame structure, Hearing Technology, Physical layer, Uplink",

author = "Abhijeet Bishnu and Ankit Gupta and Mandar Gogate and Kia Dashtipour and Ahsan Adeel and Amir Hussain and Mathini Sellathurai and Tharmalingam Ratnarajah",

note = "Funding Information: This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COGMHEAR (Grant reference EP/T021063/1). Funding Information: VI. ACKNOWLEDGMENT This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COG-MHEAR (Grant reference EP/T021063/1). Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE International Conference on E-health Networking, Application and Services, HealthCom 2022 ; Conference date: 17-10-2022 Through 19-10-2022",

year = "2022",

month = dec,

day = "21",

doi = "10.1109/HealthCom54947.2022.9982772",

language = "English",

pages = "75--80",

booktitle = "2022 IEEE International Conference on E-Health Networking, Application and Services",

publisher = "IEEE",

address = "United States",

}

Bishnu, A, Gupta, A, Gogate, M, Dashtipour, K, Adeel, A, Hussain, A, Sellathurai, M & Ratnarajah, T 2022, A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids. in 2022 IEEE International Conference on E-Health Networking, Application and Services. IEEE, pp. 75-80, 2022 IEEE International Conference on E-health Networking, Application and Services, Genoa, Italy, 17/10/22. https://doi.org/10.1109/HealthCom54947.2022.9982772

TY - GEN

T1 - A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

AU - Bishnu, Abhijeet

AU - Gupta, Ankit

AU - Gogate, Mandar

AU - Dashtipour, Kia

AU - Adeel, Ahsan

AU - Hussain, Amir

AU - Sellathurai, Mathini

AU - Ratnarajah, Tharmalingam

N1 - Funding Information: This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COGMHEAR (Grant reference EP/T021063/1). Funding Information: VI. ACKNOWLEDGMENT This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COG-MHEAR (Grant reference EP/T021063/1). Publisher Copyright: © 2022 IEEE.

PY - 2022/12/21

Y1 - 2022/12/21

N2 - In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.

AB - In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.

KW - Audio-Visual Speech Enhancement

KW - Downlink

KW - Frame structure

KW - Hearing Technology

KW - Physical layer

KW - Uplink

UR - http://www.scopus.com/inward/record.url?scp=85146300855&partnerID=8YFLogxK

U2 - 10.1109/HealthCom54947.2022.9982772

DO - 10.1109/HealthCom54947.2022.9982772

M3 - Conference contribution

AN - SCOPUS:85146300855

SP - 75

EP - 80

BT - 2022 IEEE International Conference on E-Health Networking, Application and Services

PB - IEEE

T2 - 2022 IEEE International Conference on E-health Networking, Application and Services

Y2 - 17 October 2022 through 19 October 2022

ER -

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abstract

Conference

Keywords

ASJC Scopus subject areas

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this