ROSMI: A Multimodal Corpus for Map-based Instruction-Giving

Miltiadis Marios Katsakioris; Ioannis Konstas; Pierre-Yves Mignotte; Helen Hastie

doi:10.1145/3382507.3418861

ROSMI: A Multimodal Corpus for Map-based Instruction-Giving

Miltiadis Marios Katsakioris, Ioannis Konstas, Pierre-Yves Mignotte, Helen Hastie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present the publicly-available Robot Open Street Map Instructions (ROSMI) corpus: a rich multimodal dataset of map and natural language instruction pairs that was collected via crowdsourcing. The goal of this corpus is to aid in the advancement of state-of-the-art visual-dialogue tasks, including reference resolution and robot-instruction understanding. The domain described here concerns robots and autonomous systems being used for inspection and emergency response. The ROSMI corpus is unique in that it captures interaction grounded in map-based visual stimuli that is both human-readable but also contains rich metadata that is needed to plan and deploy robots and autonomous systems, thus facilitating human-robot teaming.

Original language	English
Title of host publication	ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction
Publisher	Association for Computing Machinery
Pages	680-684
Number of pages	5
ISBN (Electronic)	9781450375818
DOIs	https://doi.org/10.1145/3382507.3418861
Publication status	Published - 21 Oct 2020
Event	22nd ACM International Conference on Multimodal Interaction 2020 - Virtual, Online, Netherlands Duration: 25 Oct 2020 → 29 Oct 2020

Conference

Conference	22nd ACM International Conference on Multimodal Interaction 2020
Abbreviated title	ICMI 2020
Country/Territory	Netherlands
City	Virtual, Online
Period	25/10/20 → 29/10/20

Keywords

crowdsourcing
data collection
dialogue system
human-robot interaction
multimodal

ASJC Scopus subject areas

Hardware and Architecture
Human-Computer Interaction
Computer Science Applications
Computer Vision and Pattern Recognition

Access to Document

10.1145/3382507.3418861

Cite this

@inproceedings{2e20bcaac6b347b2a78e0b38028176bf,

title = "ROSMI: A Multimodal Corpus for Map-based Instruction-Giving",

abstract = "We present the publicly-available Robot Open Street Map Instructions (ROSMI) corpus: a rich multimodal dataset of map and natural language instruction pairs that was collected via crowdsourcing. The goal of this corpus is to aid in the advancement of state-of-the-art visual-dialogue tasks, including reference resolution and robot-instruction understanding. The domain described here concerns robots and autonomous systems being used for inspection and emergency response. The ROSMI corpus is unique in that it captures interaction grounded in map-based visual stimuli that is both human-readable but also contains rich metadata that is needed to plan and deploy robots and autonomous systems, thus facilitating human-robot teaming. ",

keywords = "crowdsourcing, data collection, dialogue system, human-robot interaction, multimodal",

author = "Katsakioris, {Miltiadis Marios} and Ioannis Konstas and Pierre-Yves Mignotte and Helen Hastie",

year = "2020",

month = oct,

day = "21",

doi = "10.1145/3382507.3418861",

language = "English",

pages = "680--684",

booktitle = "ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

note = "22nd ACM International Conference on Multimodal Interaction 2020, ICMI 2020 ; Conference date: 25-10-2020 Through 29-10-2020",

}

Katsakioris, MM, Konstas, I, Mignotte, P-Y & Hastie, H 2020, ROSMI: A Multimodal Corpus for Map-based Instruction-Giving. in ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, pp. 680-684, 22nd ACM International Conference on Multimodal Interaction 2020, Virtual, Online, Netherlands, 25/10/20. https://doi.org/10.1145/3382507.3418861

ROSMI: A Multimodal Corpus for Map-based Instruction-Giving. / Katsakioris, Miltiadis Marios; Konstas, Ioannis; Mignotte, Pierre-Yves et al.
ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, 2020. p. 680-684.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - ROSMI

T2 - 22nd ACM International Conference on Multimodal Interaction 2020

AU - Katsakioris, Miltiadis Marios

AU - Konstas, Ioannis

AU - Mignotte, Pierre-Yves

AU - Hastie, Helen

PY - 2020/10/21

Y1 - 2020/10/21

N2 - We present the publicly-available Robot Open Street Map Instructions (ROSMI) corpus: a rich multimodal dataset of map and natural language instruction pairs that was collected via crowdsourcing. The goal of this corpus is to aid in the advancement of state-of-the-art visual-dialogue tasks, including reference resolution and robot-instruction understanding. The domain described here concerns robots and autonomous systems being used for inspection and emergency response. The ROSMI corpus is unique in that it captures interaction grounded in map-based visual stimuli that is both human-readable but also contains rich metadata that is needed to plan and deploy robots and autonomous systems, thus facilitating human-robot teaming.

AB - We present the publicly-available Robot Open Street Map Instructions (ROSMI) corpus: a rich multimodal dataset of map and natural language instruction pairs that was collected via crowdsourcing. The goal of this corpus is to aid in the advancement of state-of-the-art visual-dialogue tasks, including reference resolution and robot-instruction understanding. The domain described here concerns robots and autonomous systems being used for inspection and emergency response. The ROSMI corpus is unique in that it captures interaction grounded in map-based visual stimuli that is both human-readable but also contains rich metadata that is needed to plan and deploy robots and autonomous systems, thus facilitating human-robot teaming.

KW - crowdsourcing

KW - data collection

KW - dialogue system

KW - human-robot interaction

KW - multimodal

UR - http://www.scopus.com/inward/record.url?scp=85096723114&partnerID=8YFLogxK

U2 - 10.1145/3382507.3418861

DO - 10.1145/3382507.3418861

M3 - Conference contribution

AN - SCOPUS:85096723114

SP - 680

EP - 684

BT - ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

PB - Association for Computing Machinery

Y2 - 25 October 2020 through 29 October 2020

ER -

ROSMI: A Multimodal Corpus for Map-based Instruction-Giving

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this