In 1984, while AI researchers were still grappling with the failures Minsky encountered with SNARC decades earlier, Richard Sutton emerged with a breakthrough approach to the temporal credit assignment problem. During his six-year doctoral journey at UMass Amherst with advisor Andrew Barto, Sutton resurrected and reimagined forgotten reinforcement learning concepts. His dissertation—”Temporal Assignment in Reinforcement Learning”—would become a foundational pillar in AI’s evolution.
Unlike his computer science colleagues, Sutton approached the problem through his psychology background. Rather than engineering an optimal solution, he sought to decode how animals naturally solved this learning puzzle. His undergraduate thesis, “A Unified Theory of Expectation,” had planted a seed: perhaps expectation itself was the missing element in reinforcement learning’s equation.
Sutton’s insight was elegantly counterintuitive: what if systems learned not from actual rewards, but from anticipated ones? What if an AI received reinforcement not when it succeeded, but when it predicted its own success?
This led to his actor-critic architecture—a dual system where a “critic” continuously evaluates positions and predicts victory likelihood, while an “actor” makes decisions and receives feedback based on improvements in the critic’s predictions. Learning occurs not through final outcomes but through moment-to-moment shifts in expected reward.
This approach—temporal difference learning—would transform artificial intelligence forever.
Picture yourself in a checkers match. Through the first nine moves, neither player has gained a clear advantage. Then, on your tenth move, you execute a brilliant double jump that forces your opponent into a defensive position. As you survey the board, you feel that rush of recognition—you’ve just seized control of the game.
This moment of realisation is precisely when a temporal difference learning signal would strengthen that winning action.
Sutton theorised this approach could resolve the temporal credit assignment problem.
Rather than waiting for a game’s conclusion to learn, an AI could continuously evaluate and adjust its strategy. It could identify valuable moves and avoid detrimental ones throughout an extended checkers match, regardless of the final outcome.
After all, losing players often make several excellent moves, while winners frequently survive despite significant mistakes.
On the surface, Sutton’s approach seems to contain a fatal flaw: a paradoxical chicken-and-egg problem. Consider the circular dependency: the critic must predict outcomes based on the actor’s future moves, yet those moves haven’t been determined. Meanwhile, the actor relies entirely on the critic’s feedback to learn which actions to take. How can either component function when each requires the other’s competence to develop its own? It’s like two novice mountain climbers, each waiting for the other to lead the ascent—a recipe for remaining forever at base camp.
Yet when Sutton ran his simulations, something unexpected emerged. By training both components in tandem, they engaged in what could only be described as mutual bootstrapping. Initially, yes—the critic issued misguided rewards while the actor fumbled through poor decisions. But with each iteration, small improvements in one component slightly enhanced the other. These incremental gains compounded over thousands of trials until, remarkably, the system achieved a level of decision-making sophistication that belied its paradoxical foundations.
At last, an AI system was capable of making remarkably intelligent decisions.