While Sutton was developing Temporal Differences (TD) Learning, a young physicist named Gerald Tesauro was working on getting AI to play backgammon at IBM Research, the same team that would later create Deep Blue and Watson. Before the well-known chess-playing AI and the Jeopardy! contestant, there was a system called Neurogammon. This system was trained on the recorded games of numerous backgammon experts. Instead of learning through trial and error, it aimed to mimic what it “thought” a human professional would do. By 1989, Neurogammon could defeat every other backgammon-playing computer system, although it still couldn’t match a human player. Tesauro had spent several years teaching machines to outsmart humans in games, and then he came across Sutton’s work and decided to put it to the test. By 1994, his machine had reached an impressive level of performance, confirming that Sutton’s theory was effective in practice.
As discussed in earlier blog entries, reinforcement learning marked the second significant advancement in the development of biological brains. The brains of mammals are extraordinary, adept at interpreting both environmental and internal patterns, which facilitates problem-solving essential for survival. Currently, neuroscientists assert that they have pinpointed the biological basis of reinforcement, which operates through a close collaboration between the basal ganglia and the hypothalamus. TD learning is not a technological innovation, but an ancient mechanism woven into animal brains to make reinforcement learning work.
The basal ganglia is situated between the cortex and the thalamus. It receives input from the cortex, thalamus, and midbrain, allowing it to track an animal’s actions and its surrounding environment. Information navigates through a complex network of substructures within the basal ganglia, branching, merging, transforming, and rearranging until it reaches the output nucleus, which hosts thousands to millions of inhibitory neurons that establish extensive and influential connections to motor centres in the brainstem. This output nucleus is inherently active. The basal ganglia continuously regulates and deregulates specific actions, functioning as a universal director of an animal’s behaviour.
The proper functioning of the basal ganglia is crucial for human life. A hallmark symptom of Parkinson’s disease is the difficulty in starting movement. Patients may remain seated for several minutes before they can find the motivation even to sit up. This symptom arises from a disruption in the basal ganglia, resulting in a constant state of blocking all actions and preventing patients from initiating even the most basic movements.
Along with processing information about an animal’s behaviour and surroundings, the basal ganglia also obtain signals from a group of dopamine neurons. The synapses found in the basal ganglia feature various dopamine receptors, each reacting differently. These varying dopamine levels enhance or diminish particular synapses, altering the way the basal ganglia interprets input.
The basal ganglia is responsible for learning to replicate actions that enhance dopamine release. It facilitates the likelihood of actions that result in dopamine release while diminishing the probability of actions that inhibit dopamine. This process resembles Sutton’s machine learning actor model, which is structured to repeat behaviours that yield positive reinforcement and to suppress those that result in negative consequences. Essentially, the basal ganglia serves as the biological locus of reinforcement learning.
Reinforcement learning did not develop solely from the basal ganglia; instead, it arose from a long-standing interaction between the basal ganglia and another distinctly vertebrate structure known as the hypothalamus, a small region located at the base of the forebrain. In the brains of vertebrates, the hypothalamus initially regulates the release of dopamine. This structure contains valence neurons that have been inherited from the valence sensory system of ancestral bilaterians.
The hypothalamus simplifies external stimuli into categories of good and bad, initiating reflexive reactions accordingly. Its valence neurons are linked to a cluster of dopamine neurons responsible for distributing dopamine throughout the basal ganglia. The hypothalamus does not respond to predictive cues; it only reacts when it receives what it desires—food when hungry, warmth when cold. It determines actual rewards. In our metaphor of an AI playing backgammon, the hypothalamus informs the brain about whether it has won or lost the game, but it does not provide feedback on performance during the game itself.
As Marvin Minsky discovered during his efforts to create reinforcement learning algorithms in the 1950s (see our blog on SNARC), if brains were to learn solely from tangible rewards, they would be incapable of performing anything particularly intelligent. They would encounter the issue of temporal credit assignment. So, how does dopamine shift from being a valence signal for real rewards to a temporal difference signal for variations in anticipated future rewards?
A prominent theory regarding the function of the basal ganglia posits that these parallel circuits operate as a form of Sutton’s actor-critic system, facilitating temporal difference learning. One circuit acts as the actor, learning to replicate actions that lead to dopamine release, while the other serves as the critic, learning to foresee future rewards and initiate its own dopamine activation. Initially, the basal ganglian learner relies solely on feedback from the hypothalamic judge, but gradually it develops the ability to self-assess, recognising errors even before receiving input from the hypothalamus. This explains why dopamine neurons initially respond to the delivery of rewards, but as time progresses, their activation shifts toward predictive signals. Consequently, receiving a reward that was anticipated does not elicit dopamine release because predictions made by the basal ganglia diminish the excitement generated by the hypothalamus.
The circuitry of the basal ganglia, which first developed in the tiny brains of early vertebrates and has persisted for around 500 million years, seems to represent the biological embodiment of Sutton’s actor-critic system. Temporal difference learning, the structure of vertebrate basal ganglia, the characteristics of dopamine responses, the capacity to learn exact time intervals, and the ability to learn from omissions are all interconnected within the same mechanisms that facilitate trial-and-error learning.