In “Romy, Roby and the Secrets of Sleep,” there’s a moment where Roby, a quirky robot, inquires of his friend Romy, a food-loving cat: “Romy, are you intelligent?” This leads to a humorous exchange, with the entire family joining in to contribute. Is there a distinction between intelligence and wisdom? When we think about the SETI project (referring to the organised scientific endeavours aimed at detecting extraterrestrial intelligence by analysing electromagnetic signals and other possible technosignatures from space), things become even more complicated. Should we label biological beings as unintelligent if they lack the ability to send or receive a radio signal?
Intelligence is a remarkable biological phenomenon, and now we strive to create thinking machines that replicate its brilliance. We assess machine intelligence through specific tests, benchmarks, and milestones that aim to indicate when a system has evolved from ‘simple software’ to something greater. Occasionally, these challenges relate to significant events or historical milestones, such as defeating a human in chess or Go, or successfully passing the Turing test. At other times, they are benchmarks associated with completing tasks traditionally regarded as human, like achieving high scores on standardised examinations, solving logic puzzles, or writing computer code.
A well-known benchmark in artificial intelligence is the Turing test, introduced by Alan Turing in 1950 and originally termed the “imitation game.” This test provided a practical approach to circumvent the elusive question: “Can machines think?” Rather than getting bogged down in definitions, Turing proposed assessing whether a machine could engage in conversation so convincingly that an evaluator would struggle to distinguish it from a human.
Various organisations and individuals have asserted that the Turing test has been successfully passed multiple times, with the most notable instance occurring in 2014 with Eugene Goostman. This chatbot, posing as a 13-year-old Ukrainian boy, managed to deceive 33% of the judges during a five-minute conversation. As Gary Marcus observed at the time: “The winners aren’t genuinely intelligent; instead, they tend to be more like parlour tricks, and they’re almost inherently deceitful. If a person asks a machine, “How tall are you?” and the machine wants to win the Turing test, it has no choice but to confabulate. It has turned out, in fact, that the winners tend to use bluster and misdirection far more than anything approximating true intelligence.”
Developing an AI that can fully replicate human-level or general intelligence appears to be more challenging than originally anticipated when the term Artificial Intelligence was first introduced.
In 1955, a group of American scientists organised the inaugural conference aimed at discussing machine and robotic intelligence, which was held at Dartmouth College in 1956. This event, spearheaded by McCarthy, Minsky, Shannon, and Rochester, established the groundwork for many pivotal concepts that would influence the field for years to come. The attendees of the 1956 Dartmouth AI conference were strikingly optimistic about the prospect of achieving human-level intelligence, with numerous participants believing it could be realised in a relatively brief period, possibly within just a few summers.
As time passed, software developers and consultants frequently claimed their creations had successfully passed the Turing test, especially during moments of high expectation. At a panel discussion during the 2018 NACD summit in Washington DC, I found myself challenged by an Accenture partner who presented himself as an expert in advanced AI and asserted that “AI passed the Turing Test years ago.” In reality, he had never studied the topic or built a Machine Learning application.
Nowadays, the benchmark for a widely accepted test hinges on the confidence investors have in the LLM companies they’ve invested billions in. The public relations buzz appears to take precedence over the actual technology. You generate excitement with assertions about the future potential of an LLM, provide something that is clearly not AGI, and yet continue to promote it. Until you can’t anymore.
Sam Altman now calls AGI “not a particularly useful term,” and I think this viewpoint highlights the waning capabilities of LLMs rather than an intention to establish a new standard for evaluating agentic tasks.
Please don’t misunderstand me. I see LLMs as valuable and applicable in various industries and business areas. AIRA Technologies, as an example, uses LLMs to enable mobile network operators (MNOs) to query, control, analyse, and automate Radio Access Networks (RAN) using conversational language, as demonstrated in their flagship solution, RANGPT. Still, I will not compare current AIs to anything even slightly human. They are merely a tool, and when wielded by a skilled software developer with deep domain knowledge, they can be incredibly effective.
What is the significance of tests or goalposts? These benchmarks serve a dual purpose: they provide researchers with targets to strive for and offer the rest of us a means to evaluate whether progress is genuine. However, these benchmarks are not set in stone. Once AI achieves a particular goalpost, we frequently conclude that it was never a true indicator of intelligence in the first place.
I invite you to listen to the conversation between Roby and his family from Chapter Two of “Romy, Roby and the Secrets of Sleep.” Susan Ford, a British actress, is the voice behind the audiobook.
You can receive the whole book on Audible, Spotify, or my website, www.romyandroby.ai. Your kids will definitely enjoy it!