Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

Elisa Ramil  Brick; Vanesa Caballero Alonso; Conor O'Brien; Sheron Tong; Emilie  Tavernier; Amit Parekh; John-Angus Addlesee; Oliver Lemon

doi:10.1145/3462244.3481000

Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

Elisa Ramil Brick, Vanesa Caballero Alonso, Conor O'Brien, Sheron Tong, Emilie Tavernier, Amit Parekh, John-Angus Addlesee, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

8 Citations (Scopus)

283 Downloads (Pure)

Abstract

Sight-Impaired People (SIP) need assistance with food packaging - which can contain safety-critical information. Therefore, Textual Visual Question Answering (VQA) could prove critical in increasing the independence of SIP [26, 35], yet has only seen recent attention. For instance, comprehending text within an image is necessary to determine: what type of soup is in a can, how long to cook a microwave meal, when a box of eggs will expire, and whether a meal contains an ingredient they are allergic to. This handful of examples relate to a kitchen setting - a particularly challenging area for SIP. We extended the existing Aye-saacvoice assistant prototype with this task and setting in mind. We developed textual VQA components to accurately understand what a user is asking, extract relevant text from images in an intelligent manner, and to provide Natural Language answers that build upon the context of previous questions. As our system is created to be assistive, we designed it with a particular focus on privacy, transparency, and controllability. These are vital objectives that existing systems do not cover. We found that our system outperformed other VQA systems on real food packaging questions asked by SIP from the VizWiz dataset [19].

Original language	English
Title of host publication	ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
Subtitle of host publication	Montréal QC Canada October 18 - 22, 2021
Editors	Zakia Hammal, Carlos Busso, Catherine Pelachaud, Sharon Oviatt, Albert Ali Salah, Guoying Zhao
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	92-102
Number of pages	11
ISBN (Print)	978-1-4503-8481-0
DOIs	https://doi.org/10.1145/3462244.3481000
Publication status	Published - 18 Oct 2021
Event	23rd ACM International Conference on Multimodal Interaction 2021 - Montreal, Canada Duration: 18 Oct 2021 → 22 Oct 2021

Conference

Conference	23rd ACM International Conference on Multimodal Interaction 2021
Abbreviated title	ICMI 2021
Country/Territory	Canada
City	Montreal
Period	18/10/21 → 22/10/21

Keywords

allergens
sight-impaired people
textual VQA
textual extraction

ASJC Scopus subject areas

Computer Science Applications
Computer Vision and Pattern Recognition
Hardware and Architecture
Human-Computer Interaction

Access to Document

10.1145/3462244.3481000

Download PDF
© ACM 2021. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction, https://doi.org/10.1145/3462244.3481000
Accepted author manuscript, 24.2 MBLicence: Other

Cite this

Brick, E. R., Alonso, V. C., O'Brien, C., Tong, S., Tavernier, E., Parekh, A., Addlesee, J.-A., & Lemon, O. (2021). Am I Allergic to This? Assisting Sight Impaired People in the Kitchen. In Z. Hammal, C. Busso, C. Pelachaud, S. Oviatt, A. A. Salah, & G. Zhao (Eds.), ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021 (pp. 92-102). Association for Computing Machinery. https://doi.org/10.1145/3462244.3481000

Brick, Elisa Ramil ; Alonso, Vanesa Caballero ; O'Brien, Conor et al. / Am I Allergic to This? Assisting Sight Impaired People in the Kitchen. ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. editor / Zakia Hammal ; Carlos Busso ; Catherine Pelachaud ; Sharon Oviatt ; Albert Ali Salah ; Guoying Zhao. New York : Association for Computing Machinery, 2021. pp. 92-102

@inproceedings{f503b961545f4535b5eeda0b466388fa,

title = "Am I Allergic to This? Assisting Sight Impaired People in the Kitchen",

abstract = "Sight-Impaired People (SIP) need assistance with food packaging - which can contain safety-critical information. Therefore, Textual Visual Question Answering (VQA) could prove critical in increasing the independence of SIP [26, 35], yet has only seen recent attention. For instance, comprehending text within an image is necessary to determine: what type of soup is in a can, how long to cook a microwave meal, when a box of eggs will expire, and whether a meal contains an ingredient they are allergic to. This handful of examples relate to a kitchen setting - a particularly challenging area for SIP. We extended the existing Aye-saacvoice assistant prototype with this task and setting in mind. We developed textual VQA components to accurately understand what a user is asking, extract relevant text from images in an intelligent manner, and to provide Natural Language answers that build upon the context of previous questions. As our system is created to be assistive, we designed it with a particular focus on privacy, transparency, and controllability. These are vital objectives that existing systems do not cover. We found that our system outperformed other VQA systems on real food packaging questions asked by SIP from the VizWiz dataset [19].",

keywords = "allergens, sight-impaired people, textual VQA, textual extraction",

author = "Brick, {Elisa Ramil} and Alonso, {Vanesa Caballero} and Conor O'Brien and Sheron Tong and Emilie Tavernier and Amit Parekh and John-Angus Addlesee and Oliver Lemon",

year = "2021",

month = oct,

day = "18",

doi = "10.1145/3462244.3481000",

language = "English",

isbn = "978-1-4503-8481-0 ",

pages = "92--102",

editor = "Zakia Hammal and Carlos Busso and Pelachaud, {Catherine } and Oviatt, {Sharon } and Salah, {Albert Ali } and Zhao, {Guoying }",

booktitle = "ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

note = "23rd ACM International Conference on Multimodal Interaction 2021, ICMI 2021 ; Conference date: 18-10-2021 Through 22-10-2021",

}

Brick, ER, Alonso, VC, O'Brien, C, Tong, S, Tavernier, E, Parekh, A, Addlesee, J-A & Lemon, O 2021, Am I Allergic to This? Assisting Sight Impaired People in the Kitchen. in Z Hammal, C Busso, C Pelachaud, S Oviatt, AA Salah & G Zhao (eds), ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. Association for Computing Machinery, New York, pp. 92-102, 23rd ACM International Conference on Multimodal Interaction 2021, Montreal, Quebec, Canada, 18/10/21. https://doi.org/10.1145/3462244.3481000

Am I Allergic to This? Assisting Sight Impaired People in the Kitchen. / Brick, Elisa Ramil ; Alonso, Vanesa Caballero; O'Brien, Conor et al.
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. ed. / Zakia Hammal; Carlos Busso; Catherine Pelachaud; Sharon Oviatt; Albert Ali Salah; Guoying Zhao. New York: Association for Computing Machinery, 2021. p. 92-102.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

AU - Brick, Elisa Ramil

AU - Alonso, Vanesa Caballero

AU - O'Brien, Conor

AU - Tong, Sheron

AU - Tavernier, Emilie

AU - Parekh, Amit

AU - Addlesee, John-Angus

AU - Lemon, Oliver

PY - 2021/10/18

Y1 - 2021/10/18

N2 - Sight-Impaired People (SIP) need assistance with food packaging - which can contain safety-critical information. Therefore, Textual Visual Question Answering (VQA) could prove critical in increasing the independence of SIP [26, 35], yet has only seen recent attention. For instance, comprehending text within an image is necessary to determine: what type of soup is in a can, how long to cook a microwave meal, when a box of eggs will expire, and whether a meal contains an ingredient they are allergic to. This handful of examples relate to a kitchen setting - a particularly challenging area for SIP. We extended the existing Aye-saacvoice assistant prototype with this task and setting in mind. We developed textual VQA components to accurately understand what a user is asking, extract relevant text from images in an intelligent manner, and to provide Natural Language answers that build upon the context of previous questions. As our system is created to be assistive, we designed it with a particular focus on privacy, transparency, and controllability. These are vital objectives that existing systems do not cover. We found that our system outperformed other VQA systems on real food packaging questions asked by SIP from the VizWiz dataset [19].

AB - Sight-Impaired People (SIP) need assistance with food packaging - which can contain safety-critical information. Therefore, Textual Visual Question Answering (VQA) could prove critical in increasing the independence of SIP [26, 35], yet has only seen recent attention. For instance, comprehending text within an image is necessary to determine: what type of soup is in a can, how long to cook a microwave meal, when a box of eggs will expire, and whether a meal contains an ingredient they are allergic to. This handful of examples relate to a kitchen setting - a particularly challenging area for SIP. We extended the existing Aye-saacvoice assistant prototype with this task and setting in mind. We developed textual VQA components to accurately understand what a user is asking, extract relevant text from images in an intelligent manner, and to provide Natural Language answers that build upon the context of previous questions. As our system is created to be assistive, we designed it with a particular focus on privacy, transparency, and controllability. These are vital objectives that existing systems do not cover. We found that our system outperformed other VQA systems on real food packaging questions asked by SIP from the VizWiz dataset [19].

KW - allergens

KW - sight-impaired people

KW - textual VQA

KW - textual extraction

UR - http://www.scopus.com/inward/record.url?scp=85118966891&partnerID=8YFLogxK

U2 - 10.1145/3462244.3481000

DO - 10.1145/3462244.3481000

M3 - Conference contribution

SN - 978-1-4503-8481-0

SP - 92

EP - 102

BT - ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

A2 - Hammal, Zakia

A2 - Busso, Carlos

A2 - Pelachaud, Catherine

A2 - Oviatt, Sharon

A2 - Salah, Albert Ali

A2 - Zhao, Guoying

PB - Association for Computing Machinery

CY - New York

T2 - 23rd ACM International Conference on Multimodal Interaction 2021

Y2 - 18 October 2021 through 22 October 2021

ER -

Brick ER, Alonso VC, O'Brien C, Tong S, Tavernier E, Parekh A et al. Am I Allergic to This? Assisting Sight Impaired People in the Kitchen. In Hammal Z, Busso C, Pelachaud C, Oviatt S, Salah AA, Zhao G, editors, ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. New York: Association for Computing Machinery. 2021. p. 92-102 doi: 10.1145/3462244.3481000

Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Dataset supporting the paper "Am I Allergic to This? Assisting Sight Impaired People in the Kitchen"

Cite this

Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Datasets

Dataset supporting the paper "Am I Allergic to This? Assisting Sight Impaired People in the Kitchen"

Cite this