Multimodal Intelligent Assistance with Vision, Language and Speech for Enhanced Assistive Technology for the Visually Impaired and Elderly

Navigation assistance is essential for visually impaired and elderly individuals, as traditional tools often lack the necessary feedback for safe and independent mobility. A smart navigation system that integrates text-to-speech (TTS) and real-time scene analysis technology is introduced in this paper to assist visually impaired and elderly individuals with safe navigation. The system utilizes optical character recognition (OCR) and Tesseract to extract and read text from the environment, specifically medicine labels. Additionally, the system uses YOLOv8 for object detection to identify and describe the user's surroundings. The detected objects are passed to Bootstrapping Language-Image Pre-training (BLIP) for scene captioning, which is then converted into speech through the TTS module. The system provides real-time auditory feedback, offering guidance on both objects and text, thereby enhancing mobility and safety. Experimental results demonstrated a TTS word error rate (WER) of 9.2% and a scene recognition accuracy of 92.6%. The efficacy of the system is demonstrated in this paper through its ability to provide reliable and informative navigation support.