Overview

The Problem: Traditional text-entry methods fail individuals with motor impairments or those requiring touch-free interaction. Existing mid-air "air-writing" systems are often slow, physically exhausting, and suffer from high error rates due to human tremor and spatial uncertainty.

The Solution: I engineered a gesture-based predictive keyboard that replaces complex air-writing with a "directional intent" model. By mapping 26 letters to just 8 core directional gestures and utilizing a Hidden Markov Model (HMM) paired with a Viterbi decoder, the system transforms noisy, tremor-prone hand movements into accurate, context-aware text.

The Impact: This project bridges the gap between raw human movement and digital intent. By mathematically filtering micro-tremors and leveraging linguistic probability, it creates an accessible, low-latency interface that achieves significantly higher accuracy than path-based approaches, proving that intelligent software can compensate for physical hardware limitations.

Technical Stack & Challenges

Tech Stack:

• Languages: Python
• Frameworks/Tools: OpenCV, MediaPipe, NumPy, NLTK, PyAutoGUI
• Logic/Algorithms: Hidden Markov Model (HMM), Viterbi Decoding, Hill Climbing Optimization, Kinematic Feature Extraction

The "Hard" Problem:

The core challenge was navigating the "Signal vs. Intent" paradox. Real-time computer vision is inherently noisy; differentiating between a deliberate "Enter" gesture and a natural hand drift required moving beyond basic thresholds. I implemented a custom Kinematic Feature Extraction pipeline based on Rubine’s statistical gesture recognition, using directness ratios to distinguish intentional arcs from tremors. Furthermore, I optimized the keyboard layout using a hill-climbing algorithm, achieving a 22.64% accuracy improvement over standard T9 layouts by mathematically minimizing the "Substitution Trap."

Demo

A demonstration of the gesture recognition pipeline, showcasing the real-time mapping of hand landmarks into directional intent and the predictive text engine in action.

Retrospective

What Would I Do Differently?

I would integrate multi-modal feedback earlier in the development cycle. My current build relies heavily on visual monitoring, which increases cognitive load. Moving forward, I would implement auditory "clicks" or spatialized tones to provide immediate confirmation of keystrokes, reducing the need for constant eye-tracking of the screen.

Most Important Lesson Learned:

The "Pivot" was my most valuable lesson. I spent early days trying to perfect 3D character recognition, only to realize that HCI research consistently shows it to be inefficient. By embracing the "Directional Input Model" instead, I learned that engineering is not about building the most complex solution; it is about choosing the architecture that best aligns with human limitations and capabilities. Solving the user's problem was far more important than the pride of a complex, but ultimately unusable, initial design.