GestureMate: An AI-Driven System for Real-Time Malayalam Sign Language and Speech Translation

Abstract

Communication barriers between deaf-mute individuals and hearing people continue to limit inclusive interaction in everyday environments such as healthcare facilities, workplaces, and educational institutions. While sign language serves as a primary means of expression for deaf-mute users, the lack of widespread sign language knowledge among hearing individuals creates significant challenges. To address this issue, this paper presents GestureMate, an AI-driven assistive system designed for real-time translation of Malayalam Sign Language into text and speech, while also converting spoken responses into readable text.The proposed system integrates computer vision–based hand gesture recognition, speech-to-text processing, and text-to-speech synthesis to enable seamless bidirectional communication. GestureMate captures hand gestures using a camera, extracts key hand landmarks, and classifies gestures into meaningful text representations. These text outputs are further converted into audible speech for hearing users. Conversely, spoken input from hearing individuals is transcribed into text, allowing deaf-mute users to understand verbal responses instantly. A distinctive feature of the system is its custom sign training module, which allows users to define personalized or regional sign variations, making the system adaptable to diverse communication needs. Additionally, GestureMate includes a sign language learning module to promote awareness and accessibility among hearing users.Experimental evaluation demonstrates that the system provides accurate gesture recognition and low-latency
speech transcription, enabling smooth real-time interaction.