Context-Aware LLM Navigation for Accessibility Applications

During this project, I led the development and integration of a cutting-edge Context-Aware Large Language Model (LLM) Navigation Interface for Accessibility Applications. This initiative aimed to address significant challenges faced by visually impaired individuals in interacting with real-world environments and assistive technologies. Existing systems, such as the AI-based Hand Navigation System (HANS), supporting a tactile bracelet guiding user’s hands to objects, and the Natural Language Interface (NLI), an app integrating HANS for supermarket shopping purposes, often relied on rigid, text-based commands and manual inputs via a Graphical User Interface (GUI), hindering intuitive and continuous interaction.

My project delivered an AI interface designed to allow users to control various accessibility applications through a centralised natural language communications interface. A strength of this interface is its context-awareness, allowing it to intelligently infer user intentions based on conversational history and real-world scenarios (for example, discerning whether a user’s request for wanting ‘coffee’ pertains to a grocery item or a physical coffee cup).

It’s architecture is designed to be as generic as possible, thus allowing it to easily integrate with various other accessibility applications by simply by defining new functionalities through plaintext documents, thereby negating the need for internal code modifications. It also allows the exchange metadata between apps (for example, listening for list of available objects from HANS), allowing it to gather additional context and follow more complex instructions. This modular architecture is built on top of state-of-the-art frameworks, primarily LangGraph (an evolution from LangChain for complex multi-agent workflows).