Talking to your phone has long been a convenient alternative to typing. With the Pixel 11, Google is extending that option to people who communicate through American Sign Language (ASL).
Ahead of its Made by Google event Wednesday evening, Google DeepMind announced SL2T, a new AI model that translates sign language directly into written text. The model powers a sign-to-text feature launching in Gboard and Live Transcribe on the Pixel 11 family.
In practice, the feature allows users to use sign language via their phone’s camera for tasks they would typically use keys to perform. That could mean signing a Google search, drafting a text or email, writing in a document, or asking Gemini a question. In Live Transcribe, a user can sign a response during an in-person conversation instead of typing it out.
The feature initially translates ASL into English. Google says it will come to more devices soon, at no additional cost, with support for additional sign languages planned for the future.
How does sign-to-text work?
Teaching a phone to understand sign language takes more than showing it a dictionary of hand gestures. ASL is its own language, with distinct grammar and meaning conveyed through movements of the hands, arms, torso, head, and face, sometimes all at once.
That means SL2T has to pay attention to the signer’s entire upper body. An on-device computer vision system tracks 130 points across the person’s face, body, and hands, essentially turning the video into a moving map of geometric coordinates. The raw footage is immediately discarded, while those coordinates are sent to Google’s servers for translation, according to the DeepMind sign language team. Google’s joint impact report adds that the company does not retain logs of what users sign or the text the system produces unless someone explicitly agrees to participate in a model evaluation study.
Once the phone has mapped those movements, SL2T then translates them directly into English. A common approach in sign-language translation has been to first convert signs into written labels known as “glosses” before translating them into text.
Mashable Light Speed
The problem with that prior approach, according to the blog post, is that “glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions. Translating directly from landmarks removes artificial vocabulary limits.” In other words, skipping that word-by-word step allows SL2T to account for facial expressions, physical positioning, and other movements that carry meaning.
Google trained SL2T on more than 100,000 hours of data spanning over 50 sign languages, with roughly one-quarter of the material in ASL. It also trained SL2T to understand one-handed signing, which is useful when someone is holding their phone with the other hand, as well as signing by left-handed users.
Still, Google says SL2T is not a replacement for an interpreter
The first version still has significant limitations. DeepMind’s report says the model can struggle with regional signs, slang, complex ASL grammar, rapid fingerspelling, and meanings conveyed primarily through facial expressions or head movements. It may also generate what Google calls “ghost text,” or words the user did not sign, when someone pauses, or another person enters the camera frame.
Its accuracy can also decline in low light, when a signer is partially outside the frame, or when there is not enough contrast between a person’s hands, face, clothing, and surroundings. The model has also not been formally evaluated on people under 18 or extensively tested across signers with motor disabilities.
For those reasons, Google and its AI Sign Language Advisory Committee describe SL2T as “strictly an assistive input tool for low-stakes communication” and does not fulfill legal requirements for reasonable accommodations. It is not designed for medical appointments, legal proceedings, classrooms, job interviews, government hearings, or other settings where an error could carry serious consequences. The report also warns institutions against using the software to replace qualified human interpreters or avoid their accessibility obligations.
Google enlisted Deaf employees, outside testers, advocacy organizations, professional interpreters, and sign language experts to guide the model’s development. The company plans to continue working with its advisory committee as it expands the technology to additional sign languages and, eventually, feature AI that can generate sign language as well.
SL2T is one of several new features arriving alongside Google’s latest hardware. The company’s Made by Google event begins Wednesday at 6 p.m. ET, with Trevor Noah hosting a presentation featuring the Pixel 11 lineup, Pixel Watch 5, and more of Google’s latest product announcements.
Follow along with Mashable’s Made by Google live blog for updates and analysis throughout the event!
