While Ivan Pavlov was decoding conditional reflexes in Russia, American psychologist Edward Thorndike approached animal learning differently, focusing on how quickly animals could escape from his ingeniously designed “puzzle boxes.”
Thorndike built various cages containing unique puzzles that, when solved, would release a door. After each successful escape, he meticulously recorded the time, then placed the animal back inside for repeated trials.
Initially, Thorndike aimed to investigate imitation as a learning mechanism across species. He described behaviours using terms like “satisfying” and “discomforting” – concepts later reframed by psychologists as “rewarding” and “punishing” stimuli.
Thorndike’s work essentially pioneered reinforcement learning through trial and error. The principle seemed straightforward: strengthen behaviours with positive outcomes, discourage those with negative results.
Decades later, AI researchers attempted to implement these seemingly obvious principles from behavioural science into machines. Only then did they discover that teaching computers through reinforcement proved far more complex than training chickens or cats.
In 1951, Marvin Minsky, a Princeton doctoral student, created the world’s first reinforcement learning algorithm amid the initial surge of AI enthusiasm. This breakthrough followed a decade of foundational developments: Norbert Wiener had established intelligence as a formal field of study in his seminal “Cybernetics,” while Alan Turing had mathematically defined general-purpose computing machines. Turing, alongside economic statistician David Champernowne (who had supplied Churchill with crucial wartime data), had even developed Turochamp, a pioneering chess-playing program. Meanwhile, scientists across disciplines were beginning to decode the brain’s biological mechanisms and study neurons.
Inspired by Thorndike’s animal experiments, Minsky developed the Stochastic Neural Analogue Reinforcement Calculator (SNARC). This forty-neuron system attempted to navigate mazes using numerical feedback, mirroring how Thorndike’s cats learned to escape puzzle boxes for food rewards.
The results proved disappointing. While SNARC showed improvement in simple mazes through repeated trials, it collapsed when confronting even marginally more complex challenges. Minsky’s crucial insight was recognising the fundamental flaw in directly applying Thorndike’s reinforcement principles—rewarding positive outcomes and punishing negative ones—to machine learning algorithms.
Let me explain the reason behind this failure.
Imagine teaching a computer to play checkers through Thorndike’s trial-and-error approach. The machine begins with random moves, receiving rewards for victories and penalties for defeats. Logic suggests that after sufficient games, performance should improve. Yet a fundamental challenge emerges: feedback comes only after dozens or even hundreds of moves have determined the game’s outcome. When victory arrives, which specific decisions deserve recognition? After defeat, which choices warrant correction?
This dilemma represents what became known as the credit assignment problem.
Minsky recognised that reinforcement learning required a method for attributing success or failure to specific actions taken much earlier—what he termed the “temporal credit assignment problem.”
Unlike the sophisticated biological mechanisms we previously explored in nematodes and early vertebrates—where neuronal connections employ elegant solutions like overshadowing, latent inhibition, and blocking—Minsky’s rudimentary algorithm lacked these evolutionary advantages. His system faltered precisely because it couldn’t solve this temporal puzzle that nature had addressed through millions of years of biochemistry.
The AI community would need decades more to crack this seemingly insurmountable problem.