The system uses two implanted electrode arrays that record the brain activity produced by thinking about writing letters. This information is then collected and processed in real time by a computer, which converts that data into words on a screen.
Shenoy lab/Erika Woodrum
Humans have long been fascinated with the idea of someone or something that can read our minds — be it a telepath, a computer or Santa Claus. Now scientists say they’ve developed a system that combines machine learning and a brain-computer interface, or BCI, to read handwriting that takes place in the brain rather than on paper.
A team of scientists worked with a 65-year-old man paralyzed from the neck down, using sensors implanted in his brain to detect neural activity linked to writing by hand. As the volunteer imagined writing letters, that activity was fed to an algorithm that translated it, in real time, to text displayed on a screen.
The details of the experiment are laid out in a report in the current issue of the journal Nature. Study co-author Krishna Shenoy, a Howard Hughes Medical Institute investigator at Stanford University, says this method appears to be more potentially powerful than similar studies that have attempted to translate speech rather than handwriting.
“Right now, other investigators can achieve about a 50-word dictionary using machine learning methods when decoding speech,” Shenoy said in a statement. “By using handwriting to record from hundreds of individual neurons, we can write any letter and thus any word which provides a truly ‘open vocabulary’ that can be used in most any life situation.”
The hope is such a system could allow people who are paralyzed to type and communicate efficiently without using their hands. Elon Musk-backed startup Neuralink is famously working on similar brain implant technology.
“Just think about how much of your day is spent on a computer or communicating with another person,” Shenoy said. “Restoring the ability of people who have lost their independence to interact with computers and others is extremely important, and that is what is bringing projects like this one front and center.”
The system has been demonstrated as a proof of concept in just a single patient so far. It builds off previous work done by Shenoy’s team in which participants implanted with neural sensors attempted arm movements to move a cursor on a screen to point and click on letters and spell out words and sentences. This method allowed people to “type” around 40 characters per minute, whereas the new handwriting method saw rates of about 90 characters per minute, believed to be a new record for typing via BCI.
The researchers note that rate is about the same as someone of a similar age as the 65-year-old participant typing on a smartphone.
The system captured letters from the brain of a participant with paralysis as he imagined writing them.
F. Willett et al./Nature 2021/Howard Hughes Medical Institute
“Communication is central to how we function in society,” said Debara Tucci, director of the National Institute on Deafness and Other Communication Disorders, one of the funders for the study. “In today’s world of internet-based communication, people with severe speech and physical impairments can face significant communication barriers and, potentially, isolation. We hope these findings will encourage commercial development of this latest BCI technology.”
Shenoy would like to see handwriting integrated into a more complex BCI-based system that also allows for decoding speech and point and click navigation like that of current smartphone interfaces.
“Having those two or three modes and switching between them is something we naturally do,” he said.
Next the team aims to work with a participant who is unable to speak or suffers from a degenerative neurological disorder like amyotrophic lateral sclerosis that can rob sufferers of their speech.
