This is just cool. Researchers can explore complex neuroimaging data by “seeing” it in audiovisual format.
The video below, with its accompanying musical soundtrack, demonstrates and interprets what goes on in the brain when performing certain behaviors. A research team at Columbia University presented the technique in the open-access journal PLOS ONE.
We can now record multiple components of activity in an awake brain in real time. Brain activity research produces large volumes of data that can be difficult to explore and gain insights into the biological mechanisms behind the patterns.
The team developed a flexible toolkit enabling the translation of different types of brain imaging data—and accompanying video recordings of lab animal behavior—into audiovisual representations. The new technique was shown in three different experimental settings, demonstrating how audiovisual representations can be prepared with data from various brain imaging approaches, including 2D wide-field optical mapping (WFOM) and 3D swept confocally aligned planar excitation (SCAPE) microscopy.
Credit: Adapted from Thibodeaux et al., 2024, PLOS ONE, CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
The toolkit was applied to previously collected WFOM data that detected both neural activity and brain blood flow changes in mice during different activities. Neuronal data was represented by piano sounds that struck in time with spikes in brain activity, with the volume of each note indicating the magnitude of activity and its pitch indicating the location in the brain where the activity occurred. Blood flow data is represented by violin sounds. The sounds demonstrate the relationship between neuronal activity and blood flow.
Researchers say that their toolkit is not a substitute for quantitative analysis of neuroimaging data, but it could help scientists screen large datasets for patterns that might otherwise go unnoticed.
According to the authors: “Listening to and seeing representations of [brain activity] data is an immersive experience that can tap into this capacity of ours to recognize and interpret patterns (consider the online security feature that asks you to “select traffic lights in this image” – a challenge beyond most computers, but trivial for our brains)…[It] is almost impossible to watch and focus on both the time-varying [brain activity] data and the behavior video at the same time, our eyes will need to flick back and forth to see things that happen together. You generally need to continually replay clips over and over to be able to figure out what happened at a particular moment. Having an auditory representation of the data makes it much simpler to see (and hear) when things happen at the exact same time.”
Audiovisualization of neural activity and blood flow from the dorsal surface of the thinned skull cortex of the awake mouse.
Credit: Thibodeaux et al., 2024, PLOS ONE, CC-BY 4.0 (https://creativecommons.org/
