TY - JOUR
T1 - Visually Grounded Language Learning
T2 - a Review of Language Games, Datasets, Tasks, and Models
AU - Suglia, Alessandro
AU - Konstas, Ioannis
AU - Lemon, Oliver
N1 - Funding Information:
We would like to thank Arash Eshghi and Raquel Fernandez for their feedback on a preliminary version of this manuscript which was part of the first author’s PhD thesis (Suglia et al., 2022).
Publisher Copyright:
©2024 The Authors.
PY - 2024/1/26
Y1 - 2024/1/26
N2 - In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by “listening to the radio” only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein’s idea of ‘language games’ to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.
AB - In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by “listening to the radio” only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein’s idea of ‘language games’ to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.
UR - http://www.scopus.com/inward/record.url?scp=85184080783&partnerID=8YFLogxK
U2 - 10.1613/jair.1.15185
DO - 10.1613/jair.1.15185
M3 - Review article
AN - SCOPUS:85184080783
SN - 1076-9757
VL - 79
SP - 173
EP - 239
JO - Journal of Artificial Intelligence Research
JF - Journal of Artificial Intelligence Research
ER -