Brain Computer Interface Turns Mental Handwriting into Text on Screen
Researchers have, for the first time, decoded the neural signals associated with writing letters, then displayed typed versions of these letters in real time. They hope their invention could one day help people with paralysis communicate.
www.hhmi.org
Summary
Researchers have, for the first time, decoded the neural signals associated with writing letters, then displayed typed versions of these letters in real time. They hope their invention could one day help people with paralysis communicate.Scientists are exploring a number of ways for people with disabilities to communicate with their thoughts. The newest and fastest turns back to a vintage means for expressing oneself: handwriting.
For the first time, researchers have deciphered the brain activity associated with trying to write letters by hand. Working with a participant with paralysis who has sensors implanted in his brain, the team used an algorithm to identify letters as he attempted to write them. Then, the system displayed the text on a screen – in real time.
The innovation could, with further development, let people with paralysis rapidly type without using their hands, says study coauthor Krishna Shenoy, a Howard Hughes Medical Institute Investigator at Stanford University who jointly supervised the work with Jaimie Henderson, a Stanford neurosurgeon.
By attempting handwriting, the study participant typed 90 characters per minute – more than double the previous record for typing with such a “brain-computer interface,” Shenoy and his colleagues report in the journal Nature on May 12, 2021.
This technology and others like it have the potential to help people with all sorts of disabilities, says Jose Carmena, a neural engineer at the University of California, Berkeley, who was not involved in the study. Though the findings are preliminary, he says, “it’s a big advancement in the field.”
Brain-computer interfaces convert thought into action, Carmena says. “This paper is a perfect example: the interface decodes the thought of writing and produces the action.”
Thought-powered communication
When an injury or disease robs a person of the ability to move, the brain’s neural activity for walking, grabbing a cup of coffee, or speaking a sentence remain. Researchers can tap into this activity to help people with paralysis or amputations regain lost abilities.The need varies with the nature of the disability. Some people who have lost the use of their hands can still use a computer with speech recognition and other software. For those who have difficulty speaking, scientists have been developing other ways to help people communicate.
Two tiny arrays of implanted electrodes relayed information from the brain area that controls the hands and arms to an algorithm, which translated it into letters that appeared on a screen.
In recent years, Shenoy’s team has decoded the neural activity associated with speech in the hopes of reproducing it. They have also devised a way for participants with implanted sensors to use their thoughts associated with attempted arm movements to move a cursor on a screen. Pointing at and clicking on letters in this way let people type about 40 characters per minute, the previous speed record for typing with a brain computer interface (BCI).
No one, however, had looked at handwriting. Frank Willett, an HHMI research specialist and neuroscientist in Shenoy’s group, wondered if it might be possible to harness the brain signals evoked by putting pen to paper. “We want to find new ways of letting people communicate faster,” he says. He was also motivated by the opportunity to try something different.
The team worked with a participant enrolled in a clinical trial called BrainGate2, which is testing the safety of BCIs that relay information directly from a participant’s brain to a computer. (The trial’s director is Leigh Hochberg, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and the Providence VA Medical Center.) Henderson implanted two tiny sensors into the part of the brain that controls the hand and arm, making it possible for the person to, for example, move a robotic arm or a cursor on a screen by attempting to move their own paralyzed arm.
The participant, who was 65 years old at the time of the research, had a spinal cord injury that left him paralyzed from the neck down. Using signals the sensors picked up from individual neurons when the man imagined writing, a machine learning algorithm recognized the patterns his brain produced with each letter. With this system, the man could copy sentences and answer questions at a rate similar to that of someone his age typing on a smartphone.
This so-called “Brain-to-Text” BCI is so fast because each letter elicits a highly distinctive activity pattern, making it relatively easy for the algorithm to distinguish one from another, Willett says.
A new system
Shenoy’s team envisions using attempted handwriting for text entry as part of a more comprehensive system that also includes point-and-click navigation, much like that used on current smartphones, and even attempted speech decoding. “Having those two or three modes and switching between them is something we naturally do,” he says.As the participant imagined writing a letter or symbol, sensors implanted in his brain picked up on patterns of electrical activity, which an algorithm interpreted to trace the path of his imaginary pen. Credit: F. Willett et al./Nature 2021
Next, Shenoy says, the team intends to work with a participant who cannot speak, such as someone with amyotrophic lateral sclerosis, a degenerative neurological disorder that results in the loss of movement and speech.
The new system could potentially help those suffering from paralysis caused by a number of conditions, Henderson adds. Those include brain stem stroke, which afflicted Jean-Dominique Bauby, the author of the book The Diving Bell and the Butterfly. “He was able to write this moving and beautiful book by selecting characters painstakingly, one at a time, using eye movement,” Henderson says. “Imagine what he could have done with Frank’s handwriting interface!”
