High-performance brain-to-text communication via handwriting

  • Nature. 2021 May;593(7858):249-254. doi: 10.1038/s41586-021-03506-2.
Francis R Willett  1  2  3 ,  Donald T Avansino  4 ,  Leigh R Hochberg  5  6  7  8 ,  Jaimie M Henderson  9  10  11 ,  Krishna V Shenoy  4  12  10  11  13  14
Affiliations
  • 1. Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA. [email protected].
  • 2. Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA, USA. [email protected].
  • 3. Department of Electrical Engineering, Stanford University, Stanford, CA, USA. [email protected].
  • 4. Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA.
  • 5. VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA.
  • 6. School of Engineering, Brown University, Providence, RI, USA.
  • 7. Carney Institute for Brain Science, Brown University, Providence, RI, USA.
  • 8. Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
  • 9. Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA, USA.
  • 10. Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
  • 11. Bio-X Institute, Stanford University, Stanford, CA, USA.
  • 12. Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
  • 13. Department of Bioengineering, Stanford University, Stanford, CA, USA.
  • 14. Department of Neurobiology, Stanford University, Stanford, CA, USA.
PMID: 33981047 DOI: 10.1038/s41586-021-03506-2
Abstract

Brain-computer interfaces (BCIs) can restore communication to people who have lost the ability to move or speak. So far, a major focus of BCI research has been on restoring gross motor skills, such as reaching and grasping1-5 or point-and-click typing with a computer cursor6,7. However, rapid sequences of highly dexterous behaviours, such as handwriting or touch typing, might enable faster rates of communication. Here we developed an intracortical BCI that decodes attempted handwriting movements from neural activity in the motor cortex and translates it to text in real time, using a recurrent neural network decoding approach. With this BCI, our study participant, whose hand was paralysed from Spinal Cord Injury, achieved typing speeds of 90 characters per minute with 94.1% raw accuracy online, and greater than 99% accuracy offline with a general-purpose autocorrect. To our knowledge, these typing speeds exceed those reported for any other BCI, and are comparable to typical smartphone typing speeds of individuals in the age group of our participant (115 characters per minute)8. Finally, theoretical considerations explain why temporally complex movements, such as handwriting, may be fundamentally easier to decode than point-to-point movements. Our results open a new approach for BCIs and demonstrate the feasibility of accurately decoding rapid, dexterous movements years after paralysis.