2019 Seed Grants

Investigators

• Darcy Kelley, Harold Weintraub Professor of Biological Sciences, Columbia University
• Ursula Kwong-Brown, Senior Research Staff, Kelley Laboratory, Columbia University

Description

Throughout history, humans have developed tools in order to increase their range of control over their environment for a variety of needs. Tools can facilitate the connection between intention and reality. Tools elevate the abilities of novices and those with physical or other limitations. They can also provide superhuman abilities and new opportunities for creativity and refinement in expert practitioners. Tools are inherently linked to some of the earliest documented evidence of human artistic expression. What connects intention, such as the desire to create art, with the human body’s physical abilities to produce art? What if intentionality were to bypass the limitations of our peripheral nervous system and act directly upon the world around us? What if we could produce art in the form of music without moving a muscle?

Using a new neurotechnology that collects and processes large amounts of electromyography (EMG) data, it is possible to measure tiny signals coming from the peripheral nervous system in real-time. When these motor signals are large enough, they activate muscles to move different parts of our body. The CTRL-labs wristband, developed by recent Columbia University Neuroscience PhD graduates, is a novel, highly-sensitive EMG sensor that can capture these subthreshold signals and reappropriate them for alternative uses. Performing a simple action, such as flexing a finger, activates several motor units (a collection of muscle fibers activated by the same peripheral nerve) within a muscle in the forearm. However, just thinking about moving a finger still activates a few motor units, although the finger won’t actually move. The EMG sensors built into the wristband can track over 100 different motor units at a time and can pick up these subthreshold signals to turn thoughts into actions. This activity is detected by the wristband and reappropriated into discrete electrical signals that can be programmed into a computer interface that can ultimately control anything — from the movement of a hand avatar on a computer screen to the production of sound and light (Melcer, Astolfi, Remaley, Berenzweig, & Giurgica-Tiron, 2018). Effectively, the neural activity directly controls the artistic output, bypassing the physical limitations of the hand. 

Darcy Kelley, Harold Weintraub Professor of Biological Sciences, and musician-researcher Ursula Kwong-Brown propose to use this device to create a multimodal instrument that generates and spatializes musical sounds. Over the course of one year, they will develop, program, and pilot the wristband into an interface that can be used to control multiple musical variables (instrument type, ranges of notes, keys, chords, etc.) with limited bodily movement. 

This multimodal instrument project will help to establish new connections between neuroscience and electronic music by exploring the boundaries of sensorimotor control of actions created by the brain. A multimodal instrument that harnesses “intentions” to control specific musical outcomes could revolutionize thinking about how mind and thought generate effects on the physical world. By connecting motor neuron activity directly to external actions – bypassing the tendons, fingers, and general physical wiring of the body – an individual could greatly expand his/her capability and talent, from movement to music and beyond. This instrument will not only expand the expressive ability of performers but also allow individuals with limited physical capabilities to better express their artistic visions, thoughts, and emotions.