Richard Sutton had theorised about connections between temporal difference learning and brain function, but it was his colleague Peter Dayan who discovered the evidence. Working alongside fellow postdoc Read Montague at the Salk Institute in San Diego, Dayan believed that vertebrate brains must implement temporal difference learning algorithms. The two researchers began systematically searching through neuroscience literature for supporting data.
The key to understanding reinforcement learning in vertebrate brains lies in dopamine. Scientists initially interpreted dopamine as a simple pleasure signal—animals repeated behaviours that triggered dopamine release because they felt good. This aligns with Thorndike’s concept of trial-and-error learning, where animals repeat behaviours that lead to satisfying outcomes.
However, subsequent research revealed dopamine’s true role was not about “liking” but “wanting.” The breakthrough came from German neuroscientist Wolfram Schultz, who pioneered measurements of individual dopamine neurons during reward prediction tasks. In his experiments, monkeys viewed geometric shapes before receiving juice rewards.
Schultz observed a fascinating pattern: initially, dopamine neurons fired strongly when rewards arrived but showed no response to predictive cues. As monkeys learned the association between cues and rewards, the dopamine response shifted—firing at the predictive cue rather than the reward itself. When expected rewards failed to materialise, dopamine activity dropped below baseline. This pattern demonstrated that dopamine neurons encoded prediction errors—the difference between expected and actual outcomes—rather than rewards themselves.
When Dayan and Montague discovered Schultz’s data during their literature review, they experienced what scientists dream of—that moment when disparate puzzle pieces suddenly lock into place. The monkey dopamine patterns matched Sutton’s temporal difference learning algorithm. Like elegant code running in neural hardware, the system worked on three principles: dopamine spiked when cues predicted better rewards than expected, remained unchanged when rewards arrived exactly as anticipated, and dipped below baseline when expected rewards failed to materialise. This wasn’t merely recording rewards—it was calculating prediction errors across time. The implications extended far beyond laboratory monkeys. This mechanism explained why addiction forms even when substances or behaviours cease providing pleasure—the brain’s reinforcement system responds to prediction signals, not the rewards themselves.
Dayan’s and Monague’s 1997 collaboration with Schultz culminated in the paper “A Neural Substrate of Prediction and Reward,” which stands as a watershed moment in scientific history and a beautiful testimony of collaboration between AI and neuroscience.
Scientists have discovered temporal difference learning signals in vertebrate brains—from fish to rats to humans. Yet these neural computations remain absent in nematodes and other simple bilaterians. For these ancient organisms, dopamine served merely as a proximity alert: “something beneficial is close by”—evolution’s first draft of desire. The vertebrate revolution transformed this rudimentary alert system into something far more powerful: not just a trigger for wanting, but a precise mathematical calculation that could predict and evaluate rewards across time.
Temporal difference learning, disappointment, relief, and time perception form an interconnected system. Without precise time perception, a brain couldn’t distinguish between something that won’t happen and something that hasn’t happened yet—making it impossible to learn from absences, experience disappointment, or feel relief. These capabilities all emerge from the evolutionary foundation of TD learning.
Consider how disappointment and relief—emotions so fundamental we barely notice them—arise naturally from a brain evolved to predict future rewards. Even fish can be trained using not just conventional rewards and punishments, but also through the strategic absence of expected outcomes.
While many organisms possess basic timekeeping mechanisms—circadian rhythms govern bacteria, plants, and animals alike—vertebrates stand apart in their extraordinary temporal precision. This precision enables the sophisticated prediction systems that drive our learning and emotional responses.
Schulz, Dayan and Montague inspired breakthroughs in how AI systems learn from experience, adapt to environments, and optimise decision-making, seeding the development of algorithms that use feedback to improve performance in complex tasks.