How The Brain Learns To Control Itself

Behavioral adaptation to environmental changes and unexpected events is crucial for survival, and it requires efficient decision-making and learning capabilities. Neuromodulators (especially a sub-class of them, called monoamines) are molecules playing an essential role in this adaptation process. Most monoamines are synthesized by specific neurons in a brain region called the brainstem and then released toward large portions of the brain itself (like a broadcast).

Monoamines work like internal controllers, tuning our brain to optimize its performance. For example, norepinephrine (NE) is a monoamine involved in modulating the speed at which we build new memories as a response to environmental changes and in controlling the amount of physical or mental effort that we need to exert in a certain situation. Like an airliner’s autothrottle constantly controls the engine’s power, NE participates in controlling brain activity as a function of the environmental workload to make sure our performance stays good even when things get harder.

Monoamines not only work in reaction to changes but also proactively; they help to prepare for upcoming challenges. For example, if we go on a hike, viewing a steep climb promotes monoamine release, preparing our minds and bodies for the upcoming effort. Given this important role in regulating brain processes, it is not surprising that people suffering from mental disorders often have abnormal levels of certain monoamines, suggesting that they may play a part in the development of those diseases.

If monoamines are so important to control our brain, how does the brain control them? How does the brain know the right moment (and how much) to increase monoamines’ release? Is the disruption of monoamine control a key factor for understanding psychiatric disorders?

The medial prefrontal cortex, the grey eminence behind monoamines control

Important clues to answer these questions come from experimental studies about a brain area called the medial prefrontal cortex (MPFC). MPFC is part of the frontal lobe (the brain area crucial for planning and reasoning), and its neurons encode information that is important for making decisions, such as potential benefits (e.g. available food) and necessary costs (e.g. effort required to get it, risk, uncertainty). This allows animals to make effective decisions to improve their fitness. The MPFC is densely connected with the brainstem, the brain area where most monoamines are synthesized and released, and recent studies indicate that MPFC is involved in regulating brain states like monoamines do. For example, MPFC activity is increased when the organism has to push on its internal throttle to overcome a difficult situation.

This empirical evidence inspired our hypothesis that MPFC and the brainstem may be part of an integrated system, working together to determine the ideal level of monoamines required in any given situation. This synergy would allow controlling behavior (making the best choices to maximize benefit while minimizing cost) by controlling internal brain states (e.g. by making sure sufficient neural resources are recruited to successfully carry out the chosen behaviors). Essentially, we propose that monoamine control can be seen as a decision-making process: the brain deciding how much to control itself.

Computational modeling for understanding monoamines control

In recent years, neuroscientific research has made progress in using mathematical models to represent brain dynamics. Neuro-computational models are mathematical representations, based on real data, that describe the local behavior of brain circuits (groups of neurons or brain regions). When they are implemented on a computer program, they allow us to simulate how the neural system would behave in different conditions. These simulations provide complex insights into brain function that would be impossible to obtain by intuition.

In a recent study published in PloS Computational Biology, we used computational modeling to investigate our hypothesis that MPFC and brainstem are an integrated system that can control monoamines. We designed a neuro-computational model, named Reinforcement Meta-Learner (RML). This model is an AI software that can learn from its errors and tries to maximize its performance while minimizing the costs, like an animal would do. The RML is neurobiologically plausible, i.e. it is built based on the functional and anatomical features of the brain, and it contains artificial representations of MPFC neurons and monoamine-releasing brainstem neurons. Finally, the RML simulates both neural dynamics (the activity of neurons) and behavior (for example, choices in a given situation) during interaction with the environment. Therefore, the RML can help to understand the link between neuronal dynamics and actual behavior.

There is no boss, just a dialogue between different brain circuits

With the RML, we propose that the problem of monoamine control can be solved only by considering that the MPFC and the brainstem talk to each other, generating reciprocal influence. This dialogue implies that the brain not only learns to generate new behaviors, but it also learns how to control its own internal states, a process called meta-learning.

Although tightly interacting, the artificial MPFC and brainstem play different roles. The MPFC monitors both environment and actions by comparing expectations related to actions (e.g., biting into a shiny red apple) and their real outcomes (e.g., a rotten taste in your mouth). This comparison generates an error signal, called prediction error, which is necessary to learn action consequences and to update expectations. Based on these expectations, the RML can select new actions (decision-making) for improving its behavior (e.g., peeling the apple before taking a bite). Importantly, the artificial MPFC exploits the same self-ameliorative process to modulate monoamines, by sending control signals to the artificial brainstem. At the same time, the variation of monoamines release by the artificial brainstem influences the activity of the artificial MPFC itself. For example, monoamines control the willingness to invest more energy in a task, or the extent to which the MPFC should consider new events relevant and learn from them.

The agreement between the artificial MPFC and brainstem on when and how much monoamines must be released depends on environmental situations, as the whole system looks for the optimal solution to maximize resources intake while minimizing energy expenditure. In other words, the RML is an optimizer: it tries to maximize gains while minimizing costs.

To understand whether the RML is a good model for the real MPFC-brainstem, we had it interact with many different environments, simulating those used in actual neuroscience studies in humans and nonhuman animals. The model’s behavior was very similar to real subjects’ behavior. The neural dynamics simulated in the model was also very similar to MPFC and brainstem from real subjects. These results confirmed our hypothesis on monoamines regulation and indicated that the RML is a plausible model of how the brain makes cost-benefit decisions and learns to adapt to new challenges. Moreover, the RML provides an integrative explanation on behaviors and brain function that, until now, had been studied separately.

Future applications to study mental disorders

Abnormal monoamine levels have long been associated with mental illness. Despite the constant effort from researchers, no satisfying theory has been proposed about the role that monoamines play in mental disorders and how this role can be modulated by the environment. Our computational model simulates how monoamines are regulated and how the interaction subject-environment influences this regulation. Thus, we can use the model to investigate the origin of abnormal monoamine levels and the consequences on mental health. For example, the RML easily simulates one of the most common psychiatric symptoms: apathy. The RML simulated low energy in behavior, effort avoidance, and reduced motor activity (the main components of apathy) due to prolonged catecholamines (a type of monoamines) down-regulation.

In our simulations, we showed how this down-regulation can emerge from the interaction between biological factors (e.g. a congenital low level of neurotransmitters) and environmental factors (for example negative life events). So far, the RML simulated the control of two specific monoamines: dopamine and norepinephrine. In future research, we will apply the same computational principle of meta-learning to explain the control of another monoamine, important in the development of mood disorders and still poorly understood: serotonin.

With a complete model of monoamines control, we will be able to test the RML predictions (at the neural and behavioral levels) with both healthy volunteers and psychiatric patients, coupling behavioral experiments with neuroimaging techniques. In this way, we could obtain a quantitative theory describing the functional dynamics and the events cascade leading to specific psychiatric symptoms. Even more importantly, by comparing the RML functions with those of the brain of single patients, we may be able to obtain personalized descriptions of dysfunctional brain dynamics (a method called computational phenotyping), paving the way for personalized pharmacological and behavioral therapies.

These findings are described in the article entitled Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner, recently published in the journal PLoS Computational Biology. This work was conducted by Massimo Silvetti, Eliana Vassena, Elger Abrahamse, and Tom Verguts from Ghent University and the Italian National Research Council (CNR).