Image Colorization using CycleGAN with semantic and spatial rationality

The goal of image colorization is to make the generated color images closely approximate the color layout of the real color images. However, most of the existing methods do not consider the semantic and spatial rationality of the generated images, and this could lead to a large difference between the colored image and the real situation. In this research we propose SS-CycleGAN, a novel CycleGAN based solution for automatic image colorization. SS-CycleGAN ensures the rationality of colored images considering three aspects: high-level semantics, detailed semantics, and spatial information of the objects to be colored in the image. We designed a patch discriminator for SS-CycleGAN based on a self-attention mechanism. The self-attention mechanism can guide the patch discriminator to pay attention to spatial structure information and the semantic rationality of colored objects. The loss function of SS-CycleGAN is added with a term for detail loss, which can ensure the consistency of the details of the original image and the generated image. To extract multi-scale features of local areas to capture the spatial information of colored objects, we designed a Multi-scale Cascaded Dilated Convolution (MCDC) module. We trained and tested the proposed SS-CycleGAN on Natura Color Dataset and Flower dataset. The experimental results show that SS-CycleGAN can obtain higher quality colorized images than several state-of-the-art methods.