Two weekends ago, I had the incredible opportunity to attend my first hackathon — Caltech's HackTech 2025 — and help build something close to my heart: HandsIn, an AI-powered ASL (American Sign Language) training platform.
In 2019, my mom was diagnosed with hearing loss. Communication quickly became a daily challenge for her — and for our family. After two years, when she finally received hearing aids, life began to feel “normal” again. But that experience stayed with me. As she became an advocate for the Deaf and hard-of-hearing community, I found myself listening more, learning more, and wanting to do more.
That’s where HandsIn began.
The Problem We Tackled
While many tools focus on translating ASL into English, very few help people understand ASL — its grammar, structure, and cultural richness. This gap has real consequences in hospitals, classrooms, and elder care, where miscommunication can be not just frustrating, but dangerous.
We set out to create a safe and flexible training tool that empowers people — from healthcare providers to students and emergency responders — to practice interacting with ASL in a more authentic way.
What We Built 🚀
HandsIn allows learners to engage with an AI-powered ASL avatar that dynamically responds to their input. Instead of just translating words, the system helps users practice communication in context.
Highlights:
- 🖐️ Built a custom neural network to classify hand signs in real time
- 🗣️ Developed a pipeline to translate chatbot responses into ASL Gloss
- 🎥 Generated ASL avatar videos using a clever workaround (since the API wasn’t available!)
- ⚙️ Tech stack included Python, Streamlit, MediaPipe, DeepMotion, Gemini AI, and more
What We Learned 💡
Hackathons are tough — especially when your avatar API requires manual approval on a Saturday night 😅. But with persistence, teamwork, and some creative automation, we pushed through.
The biggest takeaway wasn’t just the code, but the community: teammates, mentors, and peers who were eager to share knowledge and help each other succeed.
Special thanks to my amazing teammates: Defne Meric Erdogan, Elena Loucks, and Nusret Efe Ucer.
Looking Ahead
This is just the beginning for HandsIn. We plan to expand with:
- Domain-specific lessons (like healthcare training)
- Richer gesture recognition
- Content highlighting ASL history and culture
Grateful for an unforgettable weekend of learning, hacking, and connection.
👉 Check out our project here: HandsIn on Devpost