Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface

David A. Robb, Francisco Javier Chiyah Garcia, Atanas Laskov, Xingkun Liu, Pedro Patron, Helen Hastie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Autonomous systems are designed to carry out activities in remote, hazardous environments without the need for operators to micro-manage them. It is, however, essential that operators maintain situation awareness in order to monitor vehicle status and handle unforeseen circumstances that may affect their intended behaviour, such as a change in the environment. We present MIRIAM, a multimodal interface that combines visual indicators of status with a conversational agent component.
This multimodal interface offers a fluid and natural way for operators to gain information on vehicle status and faults, mission progress and to set reminders. We describe the system and an evaluation study providing evidence that such an interactive multimodal interface can assist in maintaining situation awareness for operators of autonomous systems, irrespective of cognitive styles.
Original languageEnglish
Title of host publicationICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction
PublisherACM
Pages384-392
Number of pages9
ISBN (Print)9781450356923
DOIs
Publication statusPublished - 2 Oct 2018

Fingerprint

Fluids

Keywords

  • Multimodal output
  • natural language generation
  • autonomous systems
  • situation awareness
  • cognitive styles

Cite this

Robb, D. A., Garcia, F. J. C., Laskov, A., Liu, X., Patron, P., & Hastie, H. (2018). Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. In ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction (pp. 384-392). ACM. https://doi.org/10.1145/3242969.3242974
Robb, David A. ; Garcia, Francisco Javier Chiyah ; Laskov, Atanas ; Liu, Xingkun ; Patron, Pedro ; Hastie, Helen. / Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, 2018. pp. 384-392
@inproceedings{cdc17b51be3d4e7b8d4339438bdbf579,
title = "Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface",
abstract = "Autonomous systems are designed to carry out activities in remote, hazardous environments without the need for operators to micro-manage them. It is, however, essential that operators maintain situation awareness in order to monitor vehicle status and handle unforeseen circumstances that may affect their intended behaviour, such as a change in the environment. We present MIRIAM, a multimodal interface that combines visual indicators of status with a conversational agent component. This multimodal interface offers a fluid and natural way for operators to gain information on vehicle status and faults, mission progress and to set reminders. We describe the system and an evaluation study providing evidence that such an interactive multimodal interface can assist in maintaining situation awareness for operators of autonomous systems, irrespective of cognitive styles.",
keywords = "Multimodal output, natural language generation, autonomous systems, situation awareness, cognitive styles",
author = "Robb, {David A.} and Garcia, {Francisco Javier Chiyah} and Atanas Laskov and Xingkun Liu and Pedro Patron and Helen Hastie",
year = "2018",
month = "10",
day = "2",
doi = "10.1145/3242969.3242974",
language = "English",
isbn = "9781450356923",
pages = "384--392",
booktitle = "ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction",
publisher = "ACM",

}

Robb, DA, Garcia, FJC, Laskov, A, Liu, X, Patron, P & Hastie, H 2018, Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. in ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, pp. 384-392. https://doi.org/10.1145/3242969.3242974

Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. / Robb, David A.; Garcia, Francisco Javier Chiyah; Laskov, Atanas ; Liu, Xingkun; Patron, Pedro; Hastie, Helen.

ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, 2018. p. 384-392.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface

AU - Robb, David A.

AU - Garcia, Francisco Javier Chiyah

AU - Laskov, Atanas

AU - Liu, Xingkun

AU - Patron, Pedro

AU - Hastie, Helen

PY - 2018/10/2

Y1 - 2018/10/2

N2 - Autonomous systems are designed to carry out activities in remote, hazardous environments without the need for operators to micro-manage them. It is, however, essential that operators maintain situation awareness in order to monitor vehicle status and handle unforeseen circumstances that may affect their intended behaviour, such as a change in the environment. We present MIRIAM, a multimodal interface that combines visual indicators of status with a conversational agent component. This multimodal interface offers a fluid and natural way for operators to gain information on vehicle status and faults, mission progress and to set reminders. We describe the system and an evaluation study providing evidence that such an interactive multimodal interface can assist in maintaining situation awareness for operators of autonomous systems, irrespective of cognitive styles.

AB - Autonomous systems are designed to carry out activities in remote, hazardous environments without the need for operators to micro-manage them. It is, however, essential that operators maintain situation awareness in order to monitor vehicle status and handle unforeseen circumstances that may affect their intended behaviour, such as a change in the environment. We present MIRIAM, a multimodal interface that combines visual indicators of status with a conversational agent component. This multimodal interface offers a fluid and natural way for operators to gain information on vehicle status and faults, mission progress and to set reminders. We describe the system and an evaluation study providing evidence that such an interactive multimodal interface can assist in maintaining situation awareness for operators of autonomous systems, irrespective of cognitive styles.

KW - Multimodal output

KW - natural language generation

KW - autonomous systems

KW - situation awareness

KW - cognitive styles

U2 - 10.1145/3242969.3242974

DO - 10.1145/3242969.3242974

M3 - Conference contribution

SN - 9781450356923

SP - 384

EP - 392

BT - ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction

PB - ACM

ER -

Robb DA, Garcia FJC, Laskov A, Liu X, Patron P, Hastie H. Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. In ICMI '18 Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM. 2018. p. 384-392 https://doi.org/10.1145/3242969.3242974