P201 HoloNeV: Holographic visualization tool for neural network activity

Safeer A. Mirani1, Pirah Menon1, Rosanna Migliore2, Michele Migliore2, Beniamina Mercante2, Paolo Enrico1, Sergio MG Solinas1*


1Department of Biomedical Sciences, University of Sassari, Sassari, SARDINIA ITALY
2Institute of Biophysics, National Research Council, Palermo, Italy



*Email: smgsolinas@uniss.it

Introduction
Recent neuroscience initiatives have generated extensive data on neuroanatomy and brain function, enabling the development of detailed models of brain areas.While tools like NetPyne[1]and PyNN[2]facilitate network design on NEST[3]or NEURON[4], visualization tools are lagging behind. Emerging 3D holographic devices are bridging the digital and physical worlds instancing interactive virtual objects in the real world, i.e. Mixed Reality. Here, we introduce HoloNeV, a high-performance tool for visualizing and interacting with 3D neural network models. We validate the HoloNeV on a neuronal dataset derived from the CA1 region of the mouse hippocampus[5], encompassing the somata of pyramidal cells and interneurons divided into four neuronal layers.


Methods
We developed HoloNeV in Unity 3D (Version 2022), a game development platform designed for dynamic and high-resolution of complex animations to run on the Microsoft Hololens 2 headset using the Mixed Reality Toolkit for MR integration. We leverage on the technique of GPU instancing to enhance visualization performance, enabling the rapid rendering of numerous neuron representations simultaneously, up to 300.000 somata. We designed a custom low level code to control the data management in the GPU, a.k.a. shader, which supports GPU-accelerated rendering with stereo capabilities, transforming neuron positions and activations into an immersive visual experience.Tests were run on a workstation Intel Xeon w9-3495X with 128 GB RAM and GPU Nvidia RTX A5500.


Results
Following successful hardware integration and software development, the system was tested using the brain hippocampus dataset visualized through Microsoft HoloLens 2. The immersive 3D model allows exploration of neuronal organization in the CA1 region. The visualization includes a 3D distribution of neurons, density patterns, and layer-wise organization and is able to replay the neuronal activity from stored spike trains. While using standard state of art Unity tools the performance was 10 FPS, HoloNeV performance testing showed a mean frame rate of 97.8 FPS, ensuring a comfortable user experience in mixed reality.


Discussion

We introduce HoloNeV, a mixed-reality tool for visualizing neural network activity using an holographic headset. This system allows researchers to interact with neural networks in 3D while staying aware of their physical surroundings. Key innovations include stereo rendering optimization and fine hand-tracking for direct manipulation. Researchers can customize visualization parameters such as neuron size and density in real time. Although currently limited to representing the soma without axon and dendrite, future developments will address the full neuronal morphology, the configuration of neuronal parameters, real-time data streaming from HPC facilities, paving the way for new insights in neuroscience research.




Acknowledgements
Project IR00011 EBRAINS-Italy - Mission 4, “Istruzione e Ricerca” - Component 2, “Dalla ricerca all impresa” - Line of investment 3.1 of PNRR, Action 3.1.1 NextGeneration EU (CUP B51E22000150006) awarded to P. E. and S. S., the “FeNeL” project, PNRR M4.C2.1.1 – PRIN 2022 – No. 2022JE5SK2 – CUP G53D23000380006 awarded to S. S., Project “Numeracy in Aging (NiA) CUP J53D23017580001 awarded to P. E.
References
1. Dura-Bernal, S. et al. (2018) https://doi.org/10.1101/461137
2. Davison, A. P. (2008). https://doi.org/10.3389/neuro.11.011.2008
3. Gewaltig, M.-O., & Diesmann, M. (2007). https://doi.org/10.4249/scholarpedia.1430
4. Hines, M. (2009). https://doi.org/10.3389/neuro.11.001.2009

5. Gandolfi Daniela, et al. (2022).https://doi.org/10.1038/s41598-022-18024-y