Generating Aesthetic Based Critique For Photographs

Yong Yaw Yeo; John See; Lai Kuan Wong; Hui Ngo Goh

doi:10.1109/ICIP42928.2021.9506385

Generating Aesthetic Based Critique For Photographs

Yong Yaw Yeo, John See^*, Lai Kuan Wong, Hui Ngo Goh

^*Corresponding author for this work

School of Mathematical & Computer Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

The recent surge in deep learning methods across multiple modalities has resulted in an increased interest in image captioning. Most advances in image captioning are still focused on the generation of factual-centric captions, which mainly describe the contents of an image. However, generating captions to provide a meaningful and opinionated critique of photographs is less studied. This paper presents a framework for leveraging aesthetic features encoded from an image aesthetic scorer, to synthesize human-like textual critique via a sequence decoder. Experiments on a large-scale dataset show that the proposed method is capable of producing promising results on relevant metrics relating to semantic diversity and synonymity, with qualitative observations demonstrating likewise. We also suggest the use of Word Mover’s Distance as a semantically intuitive and informative metric for this task.

Original language	English
Title of host publication	2021 IEEE International Conference on Image Processing
Publisher	IEEE
Pages	2523-2527
Number of pages	5
ISBN (Electronic)	9781665441155
DOIs	https://doi.org/10.1109/ICIP42928.2021.9506385
Publication status	Published - 23 Aug 2021
Event	28th IEEE International Conference on Image Processing 2021 - Anchorage, United States Duration: 19 Sept 2021 → 22 Sept 2021 https://www.2021.ieeeicip.org/

Conference

Conference	28th IEEE International Conference on Image Processing 2021
Abbreviated title	2021 IEEE ICIP
Country/Territory	United States
City	Anchorage
Period	19/09/21 → 22/09/21
Internet address	https://www.2021.ieeeicip.org/

Keywords

Aesthetic quality assessment
Encoder-decoder network
Image captioning
Text synthesis
Word mover’s distance

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition
Signal Processing

Access to Document

10.1109/ICIP42928.2021.9506385

Cite this

@inproceedings{970e14206bf246a594d8213f9c8267bd,

title = "Generating Aesthetic Based Critique For Photographs",

abstract = "The recent surge in deep learning methods across multiple modalities has resulted in an increased interest in image captioning. Most advances in image captioning are still focused on the generation of factual-centric captions, which mainly describe the contents of an image. However, generating captions to provide a meaningful and opinionated critique of photographs is less studied. This paper presents a framework for leveraging aesthetic features encoded from an image aesthetic scorer, to synthesize human-like textual critique via a sequence decoder. Experiments on a large-scale dataset show that the proposed method is capable of producing promising results on relevant metrics relating to semantic diversity and synonymity, with qualitative observations demonstrating likewise. We also suggest the use of Word Mover{\textquoteright}s Distance as a semantically intuitive and informative metric for this task.",

keywords = "Aesthetic quality assessment, Encoder-decoder network, Image captioning, Text synthesis, Word mover{\textquoteright}s distance",

author = "Yeo, {Yong Yaw} and John See and Wong, {Lai Kuan} and Goh, {Hui Ngo}",

note = "Funding Information: In this paper, we propose a new approach towards generating aesthetic-based image captions by directly leveraging on aesthetic features encoded by an image aesthetic scorer, alongside factual features trained from object-based classifier. Our method is able to outperform existing works on several relevant metrics of semantic diversity and synonymity. Acknowledgement: This work is supported by Malaysia Ministry of Education FRGS Research Grant (Project No: FRGS/1/2018/ICT02/MMU/02/2). Publisher Copyright: {\textcopyright} 2021 IEEE.; 28th IEEE International Conference on Image Processing 2021, 2021 IEEE ICIP ; Conference date: 19-09-2021 Through 22-09-2021",

year = "2021",

month = aug,

day = "23",

doi = "10.1109/ICIP42928.2021.9506385",

language = "English",

pages = "2523--2527",

booktitle = "2021 IEEE International Conference on Image Processing",

publisher = "IEEE",

address = "United States",

url = "https://www.2021.ieeeicip.org/",

}

TY - GEN

T1 - Generating Aesthetic Based Critique For Photographs

AU - Yeo, Yong Yaw

AU - See, John

AU - Wong, Lai Kuan

AU - Goh, Hui Ngo

N1 - Funding Information: In this paper, we propose a new approach towards generating aesthetic-based image captions by directly leveraging on aesthetic features encoded by an image aesthetic scorer, alongside factual features trained from object-based classifier. Our method is able to outperform existing works on several relevant metrics of semantic diversity and synonymity. Acknowledgement: This work is supported by Malaysia Ministry of Education FRGS Research Grant (Project No: FRGS/1/2018/ICT02/MMU/02/2). Publisher Copyright: © 2021 IEEE.

PY - 2021/8/23

Y1 - 2021/8/23

N2 - The recent surge in deep learning methods across multiple modalities has resulted in an increased interest in image captioning. Most advances in image captioning are still focused on the generation of factual-centric captions, which mainly describe the contents of an image. However, generating captions to provide a meaningful and opinionated critique of photographs is less studied. This paper presents a framework for leveraging aesthetic features encoded from an image aesthetic scorer, to synthesize human-like textual critique via a sequence decoder. Experiments on a large-scale dataset show that the proposed method is capable of producing promising results on relevant metrics relating to semantic diversity and synonymity, with qualitative observations demonstrating likewise. We also suggest the use of Word Mover’s Distance as a semantically intuitive and informative metric for this task.

AB - The recent surge in deep learning methods across multiple modalities has resulted in an increased interest in image captioning. Most advances in image captioning are still focused on the generation of factual-centric captions, which mainly describe the contents of an image. However, generating captions to provide a meaningful and opinionated critique of photographs is less studied. This paper presents a framework for leveraging aesthetic features encoded from an image aesthetic scorer, to synthesize human-like textual critique via a sequence decoder. Experiments on a large-scale dataset show that the proposed method is capable of producing promising results on relevant metrics relating to semantic diversity and synonymity, with qualitative observations demonstrating likewise. We also suggest the use of Word Mover’s Distance as a semantically intuitive and informative metric for this task.

KW - Aesthetic quality assessment

KW - Encoder-decoder network

KW - Image captioning

KW - Text synthesis

KW - Word mover’s distance

UR - http://www.scopus.com/inward/record.url?scp=85125600854&partnerID=8YFLogxK

U2 - 10.1109/ICIP42928.2021.9506385

DO - 10.1109/ICIP42928.2021.9506385

M3 - Conference contribution

AN - SCOPUS:85125600854

SP - 2523

EP - 2527

BT - 2021 IEEE International Conference on Image Processing

PB - IEEE

T2 - 28th IEEE International Conference on Image Processing 2021

Y2 - 19 September 2021 through 22 September 2021

ER -

Generating Aesthetic Based Critique For Photographs

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this