Multimodal Machine Learning for 2D to 3D Mapping in Biomedical Atlases

B. Almogadwy, N. K. Taylor, A. Burger

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
98 Downloads (Pure)


2D to 3D image registration has a vital role in medical imaging and remains a significant challenge. It primarily relates to the use and analysis of multimodal data. We address the issue by developing a multimodal machine learning algorithm that predicts the position of a 2D slice in a 3D biomedical atlas dataset based on textual annotation and image data. Our algorithm first separately analyses images and textual information using base models and then combines the outputs of the base models using a Meta-learner model. To evaluate learning models, we have built a custom accuracy function. We tested different variants of Convolutional Neural Network architectures and different transfer learning techniques to build an optimal image base model for image analysis. To analyze textual information, we used tree-based ensemble models, namely, Random Forest and XGBoost algorithms. We applied the grid search to find optimal hyperparameters for tree-based methods. We have found that the XGBoost model showed the best performance in combining predictions from different base models. Testing the developed method showed 99.55% accuracy in predicting 2D slice position in a 3D atlas model.

Original languageEnglish
Pages (from-to)64-69
Number of pages6
JournalJournal of Image and Graphics
Issue number2
Publication statusPublished - Jun 2022


  • deep learning
  • EMAP atlas. CNN
  • image registration
  • multimodal data

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Science Applications
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Multimodal Machine Learning for 2D to 3D Mapping in Biomedical Atlases'. Together they form a unique fingerprint.

Cite this