P135 Neural dynamics of reversal learning in the prefrontal cortex and recurrent neural networks
Christopher M. Kim*1, Carson C. Chow1, Bruno B. Averbeck2
1Laboratory of Biological Modeling, NIDDK/NIH, Bethesda, MD
2Laboratory of Neuropsychology, NIMH/NIH, Bethesda, MD
3Current address: Department of Mathematics, Howard University, Washington, DC
*Email: christopher.kim@howard.edu
Introduction
In a probabilistic reversal learning task, a subject learns from initial trials that one of the two options yields reward with higher probability than the other (for instance, the high-value and the low-value options are rewarded 70% and 30% of the time, respectively). When the reward probabilities of two options are reversed at a random trial, the agent must switch its choice preference to maximize reward. Such reversal learning has been used for assessing one’s ability to adapt in a dynamically changing environment with uncertain rewards [1]. In this task, reward outcomes must be integrated over multiple trials before reversing the preferred choice, as the less favorable option yields rewards stochastically.
Methods
We investigated how cortical neurons represent integration of decision-related evidence across trials in the reversal learning task. Previous works considered attractor dynamics along a line in the state space as a neural mechanism for evidence integration [2]. However, when integrating evidence across trials, the subject must perform task-related behaviors within each trial, which could induce non-stationary neural activity. To understand the neural representation of multi-trial evidence accumulation, we analyzed the activity of neurons in the prefrontal cortex of monkeys and recurrent neural networks trained to perform a reversal learning task.
Results
We found that, in a neural subspace encoding reversal probability, its activity represented integration of reward outcomes as in a line attractor. The reversal probability activity at the start of a trial was stationary, stable and consistent with the attractor dynamics. However, during the trial, the activity was associated with task-related behavior and became non-stationary, thus deviating from the line attractor. Fitting a predictive model to neural data showed that the stationary state at the trial start served as an initial condition for launching the non-stationary activity. This suggested an extension of the line attractor model with behavior-induced non-stationary dynamics.
Discussion
Our findings show that, when performing a reversal learning task, a cortical circuit represents reversal probability, not only in stable stationary states as in a line attractor model, but also in dynamic neural trajectories that can accommodate non-stationary task-related behaviors necessary for the task. Such neural mechanism demonstrates the temporal flexibility of cortical computation and opens the opportunity for extending existing neural model for evidence accumulation by augmenting temporal dynamics.
Acknowledgements
This research was supported by the Intramural Research Program of the National Institutes of Health: the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) and the National Institute of Mental Health (NIMH). This work utilized the computational resources of the NIH HPC Biowulf cluster (https://hpc.nih.gov).
References
[1] Bartolo, R., & Averbeck, B. B. (2020). Prefrontal cortex predicts state switches during reversal learning.Neuron,106(6), 1044-1054.
[2] Mante, V., Sussillo, D., Shenoy, K. V., & Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex.nature,503(7474), 78-84.
