How This AI-Powered System Aims to Cut Communication Lags for People with Paralysis

Ann, a research participant in Dr. Edward Chang’s study in El Cerrito, Calif., is connected to computers that translate her brain signals into the speech and facial movements of an avatar as she attempts to speak. Image taken May 22, 2023.

Noah Berger

Whether we’re chatting on the phone or over dinner, typical conversation happens fast — around 150 words per minute for most exchanges in English. But for people living with paralysis who use assistive devices to share their thoughts via text, this rate is considerably slower.

Edward Chang, a neurosurgeon and chair of the Department of Neurological Surgery at the University of California, San Francisco, said users of such technology communicate at a rate closer to 10 to 20 words per minute. “There is a profound, unmet need to restore faster and more expressive communication,” he added.

Chang is a leading researcher in the neuroprosthetics field, focused on designing novel technologies which aim to restore abilities like speech or movement following brain injury. In a proof-of-concept study Chang and his colleagues published in 2023 in Nature, a woman living with paralysis named Ann used a brain-computer interface powered by artificial intelligence to translate her brain signals into on-screen text combined with a digital avatar Ann co-designed to provide spoken words and facial expressions.

The system translated those signals at a median rate of nearly 80 words of text per minute, several times faster than the up to 14 words per minute she conveys with the head-tracking assistive device she’s historically used.

The device works by picking up brain signals from a grid of sensors — called an ECoG array — placed directly at the surface of Ann’s sensorimotor cortex and superior temporal gyrus. Those regions are associated with producing movement and processing sound. The grid recorded electric activity directly from the brain’s surface and transferred this data to the system via an HDMI cable plugged into a port in her skull.

Pete Bell

Signals which would have otherwise prompted movement in Ann’s vocal tract and face simultaneously powered the output of words on the screen as well as the voice and expressions of her on-screen avatar. The voice of the avatar was based on a recording of a speech Ann gave at her wedding.

Recording from the brain’s surface offers several benefits, Chang noted, like avoiding the process of implanting electrodes inside the brain, which can cause scarring and injury. It’s also more effective than recording signals from the scalp because the skull causes interference.

The signals detected by the surface array are less clear than the ones a more invasive technique could pick up, said Marco Bonizzato, an assistant professor in the Department of Electrical Engineering at Polytechnique Montréal who wasn’t involved in the study. He likened the process to making sense of a cacophony of kids’ voices inside a classroom “as opposed to you standing outside of the room and trying to figure it out from there.” But he said the processing power of the system’s ingrained AI algorithms make up for this marginally reduced quality.

Brain activity is “noisy,” Chang noted, and his team’s device only picks up a fraction of it. But he said built-in predictive algorithms — like the ones available on a smartphone keyboard — powered by the “statistics of English” add a key layer of autocorrection to the decoding process. In other words, the system’s ability to predict probable sentence structures renders it far more accurate than if it relied solely on its own training to translate Ann’s brain signals into words.

The device was also trained on phonemes, or the few dozen distinct sounds that make up words in English. This allowed the device to work more quickly and accurately because phonemes offer the system more flexibility when decoding the speech it interpreted from Ann’s brain signals, given they’re the building blocks of every word used by speakers of the language.

Ann attempts to silently speak a sentence as the speech prosthesis system translates her brain signals into synthesized speech and the facial movements of an avatar. Image taken May 22, 2023.

Noah Berger

“You're translating the language that the brain uses of electrical activity into the language of English, or any other language you're trying to decode,” Chang said.

Communication is about more than words, Chang remarked — it's also key to “identity” and “connection with other people.” He said his team plans to make future iterations of their system completely wireless and equipped with a more sensor-dense array to increase performance, and they also want to assess how it could potentially control robotic arms.  That’s important because “speech and movement are historically the two most important brain-computer interface applications,” Bonizzato said. Although engineers have developed devices people living with paralysis can use to move parts of their body, he noted those movements are typically limited and irregular. He believes advancements like the one achieved by Chang’s team’s system will help move the field forward in coming years.

Bonizzato applauded this study’s successful use of large-language models to decode speech from brain signals in real time, and he said the research is a “strong demonstration” of what this technology can do. One of its strengths, he said, is the fact the grid the researchers used is a standard FDA-approved medical device. This route saves a lot of time which would otherwise be spent proving the safety and efficacy of brand-new technology.

UCSF clinical research coordinator Max Dougherty is pictured working with Ann. Displayed on the monitor are neural signals from her cortex. Image taken May 22, 2023.

Noah Berger

There’s about a two-second delay between Ann thinking a sentence and her avatar speaking it aloud, but Chang said his team has since reduced the lag to under a second. Bonizzato said future iterations of comparable technology could use AI to predict speech as users are crafting a sentence, rather than at the conclusion of a thought, to make the speaking process more immediate.

Regardless of how systems like this one are tweaked moving forward, Bonizzato said the road to speech neuroprostheses has been paved. Now, he added, it’s a matter of fine tuning the devices harnessing this technology and getting them into the hands of users with conditions like Ann’s — a product question, rather than a scientific one.

Can you make a device really packaged in a way that anyone with this kind of disability would be able to use it and make it really have an impact on their life, and not just close it in a laboratory at the end of the experiment?” Bonizzato said. Building new technologies with life beyond the lab in mind, then, is a key step toward ensuring everyone who needs them could one day have access.

About the Author

Bella Isaacs-Thomas

Bella is the Staff Writer and Editor for BrainFacts. She previously worked as a digital science reporter at PBS NewsHour, where she was named a fellow in the 2022-2023 class of the National Science-Health-Environment Reporting Fellowships (SHERF) and earned a 2021 D.C. Science Writers Association Newsbrief Award for writing. She graduated from the University of Michigan with a bachelor's degree in 2018.