Reported by 1 source

The short version

  • Researchers developed a surface-level brain implant that decodes both speech intent and physical gestures in real time.
  • Two participants with paralysis achieved high accuracy rates in controlling a digital avatar during conversation tests.
  • Experts note the system is currently limited to isolated phrases and movements rather than continuous interaction.

Experimental brain-computer interfaces have historically forced users with severe paralysis to choose between restoring speech or restoring movement. A new study published in Nature Neuroscience on September 14 describes a device that overcomes this binary limitation by allowing simultaneous control of both functions. The technology enables participants to operate a virtual avatar that speaks and gestures concurrently, offering a more nuanced form of communication than previous systems provided.

The research team, led by scientists at the University of California, San Francisco, focused on two individuals with distinct causes of paralysis. One participant suffered from amyotrophic lateral sclerosis, while the other experienced paralysis following a stroke. Both received surgical implants containing 253 electrodes placed directly onto the sensorimotor cortex. This region of the brain is responsible for processing sensory input and coordinating motor output. Unlike many earlier devices that penetrate deep into brain tissue, this implant rests on the surface, which researchers suggest allows for safer coverage of a wider neural area.

News Journal

During the training phase, participants attempted to produce specific verbal phrases and physical actions. The vocabulary included common greetings such as “hello” and social niceties like “nice to meet you.” For gestures, the range encompassed simple motions such as waving and clapping. The computer models were then tasked with decoding the neural signals associated with these attempts. Crucially, the system was also trained on combinations of speech and gesture, aiming to replicate the natural interplay between what people say and how they move their bodies while talking.

Testing revealed varying levels of success depending on the participant’s underlying condition. In five rounds of conversation testing, the individual with ALS achieved an average accuracy rate of 85 percent for gestures and 75 percent for speech. The participant who had suffered a stroke demonstrated perfect accuracy in both categories across three test rounds. Researchers indicated that these disparities may stem from differences in how each neurological condition affects brain function, rather than flaws in the device itself.

Samantha Brosler, a bioengineer at UCSF and co-author of the study, emphasized the importance of integrating non-verbal cues into communication aids. She noted that gestures often carry distinct meanings that words alone cannot convey. For instance, nodding while saying “maybe” communicates agreement or affirmation, whereas shaking one’s head introduces doubt or refusal. Capturing these subtleties could significantly improve the quality of interaction for people who have lost the ability to move or speak naturally.

Independent experts offered mixed assessments of the findings. Daniel Rubin, a neurologist at Massachusetts General Hospital who was not involved in the research, praised the work as a significant step toward restoring naturalistic function for paralyzed individuals. However, Christian Herff, a computational neuroscientist at Maastricht University, pointed out that the current speech decoding capabilities lag behind previous reports in the field. He also highlighted that the system currently handles only isolated sentences and gestures, whereas real-world conversation involves continuous, fluid streams of both.

The researchers acknowledge that this study serves primarily as a proof of concept. Current limitations include a restricted vocabulary and the inability to control complex, continuous movements. Future work aims to expand the range of decodable phrases and enable finer motor control, such as manipulating individual joints in a robotic hand. Brosler suggested that mastering communicative gestures is just one application; the technology could eventually allow users to perform other tasks while speaking, such as cooking with a prosthetic arm.

This development follows earlier breakthroughs in brain-computer interface technology. In March, researchers reported on a system that allowed two paralyzed individuals to type on a virtual keyboard using their thoughts, with one user achieving speeds comparable to able-bodied smartphone users. The new dual-function implant builds on this momentum by addressing the social and expressive dimensions of communication. While still in early stages, the technology represents a potential bridge toward more sophisticated robotic prosthetics and immersive digital avatars for people with severe motor impairments.

The path from laboratory proof-of-concept to clinical application remains long. The device must demonstrate reliability over extended periods and handle the unpredictability of spontaneous conversation. Additionally, the surgical nature of the implant raises questions about accessibility and risk for potential patients. Nevertheless, the ability to synchronize speech and gesture marks a distinct evolution in assistive technology, moving beyond simple text generation toward a more holistic restoration of human interaction.

Sources behind this briefing

Go to the original reporting

  • Smithsonian Magazine↗This Brain Implant Allows People With Paralysis to Speak and Gesture at the Same Time Via a Mind-Controlled Virtual Avatar