Is attention all you need in medical image analysis? A review

Giorgos Papanastasiou, Nikolaos Dikaios, Jiahao Huang, Chengjia Wang, Guang Yang

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
35 Downloads (Pure)


Medical imaging is a key component in clinical diagnosis, treatment planning and clinical trial design, accounting for almost 90% of all healthcare data. CNNs achieved performance gains in medical image analysis (MIA) over the last years. CNNs can efficiently model local pixel interactions and be trained on small-scale MI data. Despite their important advances, typical CNN have relatively limited capabilities in modelling “global” pixel interactions, which restricts their generalisation ability to understand out-of-distribution data with different “global” information. The recent progress of Artificial Intelligence gave rise to Transformers, which can learn global relationships from data. However, full Transformer models need to be trained on large-scale data and involve tremendous computational complexity. Attention and Transformer compartments (“Transf/Attention”) which can well maintain properties for modelling global relationships, have been proposed as lighter alternatives of full Transformers. Recently, there is an increasing trend to co-pollinate complementary local-global properties from CNN and Transf/Attention architectures, which led to a new era of hybrid models. The past years have witnessed substantial growth in hybrid CNN-Transf/Attention models across diverse MIA problems. In this systematic review, we survey existing hybrid CNN-Transf/Attention models, review and unravel key architectural designs, analyse breakthroughs, and evaluate current and future opportunities as well as challenges. We also introduced an analysis framework on generalisation opportunities of scientific and clinical impact, based on which new data-driven domain generalisation and adaptation methods can be stimulated.
Original languageEnglish
Pages (from-to)1398-1411
Number of pages14
JournalIEEE Journal of Biomedical and Health Informatics
Issue number3
Early online date29 Dec 2023
Publication statusPublished - Mar 2024


  • Attention
  • computed tomography
  • convo- lutional neural networks
  • magnetic resonance imaging
  • medical image analysis
  • positron emission tomography
  • retinal imaging
  • transformers

ASJC Scopus subject areas

  • Health Information Management
  • Health Informatics
  • Electrical and Electronic Engineering
  • Computer Science Applications


Dive into the research topics of 'Is attention all you need in medical image analysis? A review'. Together they form a unique fingerprint.

Cite this