Fusion of colour contrasted images for early detection of oesophageal squamous cell dysplasia from endoscopic videos in real time

Xiaohong Gao, Stephen Taylor, Wei Pang, Rui Hui, Xin Lu, Barbara Braden, Philip Allan, Tim Ambrose, Carolina Arancibia-Cárcamo, Adam Bailey, Ellie Barnes, Elizabeth Bird-Lieberman, Jan Bornschein, Oliver Brain, Jane Collier, Emma Culver, James East, Alessandra Geremia, Bruce George, Lucy HowarthKelsey Jones, Paul Klenerman, Simon Leedham, Rebecca Palmer, Fiona Powrie, Astor Rodrigues, Jack Satsangi, Alison Simmons, Simon Travis, Holm Uhlig, Alissa Walsh

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
14 Downloads (Pure)


Standard white light (WL) endoscopy often misses precancerous oesophageal changes due to their only subtle differences to the surrounding normal mucosa. While deep learning (DL) based decision support systems benefit to a large extent, they face two challenges, which are limited annotated data sets and insufficient generalisation. This paper aims to fuse a DL system with human perception by exploiting computational enhancement of colour contrast. Instead of employing conventional data augmentation techniques by alternating RGB values of an image, this study employs a human colour appearance model, CIECAM, to enhance the colours of an image. When testing on a frame of endoscopic videos, the developed system firstly generates its contrast-enhanced image, then processes both original and enhanced images one after another to create initial segmentation masks. Finally, fusion takes place on the assembled list of masks obtained from both images to determine the finishing bounding boxes, segments and class labels that are rendered on the original video frame, through the application of non-maxima suppression technique (NMS). This deep learning system is built upon real-time instance segmentation network Yolact. In comparison with the same system without fusion, the sensitivity and specificity for detecting early stage of oesophagus cancer, i.e. low-grade dysplasia (LGD) increased from 75% and 88% to 83% and 97%, respectively. The video processing/play back speed is 33.46 frames per second. The main contribution includes alleviation of data source dependency of existing deep learning systems and the fusion of human perception for data augmentation.
Original languageEnglish
Pages (from-to)64-79
Number of pages16
JournalInformation Fusion
Early online date24 Nov 2022
Publication statusPublished - Apr 2023


  • Colour contrast enhancement
  • Deep machine learning
  • Early squamous cell cancer detection
  • Endoscopic treatment
  • gastrointestinal endoscopy
  • Oesophagus cancer
  • Surveillance

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Signal Processing
  • Hardware and Architecture


Dive into the research topics of 'Fusion of colour contrasted images for early detection of oesophageal squamous cell dysplasia from endoscopic videos in real time'. Together they form a unique fingerprint.

Cite this