Image Colorization using CycleGAN with semantic and spatial rationality

Bin Li, Yi Lu, Wei Pang, Huixin Xu

Research output: Contribution to journalArticlepeer-review

26 Citations (Scopus)

Abstract

The goal of image colorization is to make the generated color images closely approximate the color layout of the real color images. However, most of the existing methods do not consider the semantic and spatial rationality of the generated images, and this could lead to a large difference between the colored image and the real situation. In this research we propose SS-CycleGAN, a novel CycleGAN based solution for automatic image colorization. SS-CycleGAN ensures the rationality of colored images considering three aspects: high-level semantics, detailed semantics, and spatial information of the objects to be colored in the image. We designed a patch discriminator for SS-CycleGAN based on a self-attention mechanism. The self-attention mechanism can guide the patch discriminator to pay attention to spatial structure information and the semantic rationality of colored objects. The loss function of SS-CycleGAN is added with a term for detail loss, which can ensure the consistency of the details of the original image and the generated image. To extract multi-scale features of local areas to capture the spatial information of colored objects, we designed a Multi-scale Cascaded Dilated Convolution (MCDC) module. We trained and tested the proposed SS-CycleGAN on Natura Color Dataset and Flower dataset. The experimental results show that SS-CycleGAN can obtain higher quality colorized images than several state-of-the-art methods.

Original languageEnglish
Pages (from-to)21641-21655
Number of pages15
JournalMultimedia Tools and Applications
Volume82
Issue number14
Early online date16 Feb 2023
DOIs
Publication statusPublished - Jun 2023

Keywords

  • Cycle-consistency adversarial network
  • Detail loss
  • Image colorization
  • Multi-scale cascaded dilated convolution
  • Self-attention

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Image Colorization using CycleGAN with semantic and spatial rationality'. Together they form a unique fingerprint.

Cite this