What are Foundation Models in AI (Part 1)?

A foundation model is a model so big, so broadly trained, and so general-purpose that everything else gets built on top of it. Think of it like the concrete slab under a skyscraper. You don’t live on the concrete slab — you live in the apartments, offices, and restaurants built on it. But if the slab cracks, everything above it cracks too. 

ChatGPT, Google Gemini, Anthropic’s Claude, Meta’s LLaMA — these are all foundation models, or products built directly on top of them. They are called “foundational” because they are trained once on an almost incomprehensibly large pile of human-generated text, images, code, and data, and then fine-tuned for specific tasks. The same underlying model can power a customer service chatbot in the morning, help a surgeon summarise medical literature at noon, and assist a novelist brainstorm plot ideas at night. That versatility is the whole point. 

How We Got Here: A Very Fast History

Twenty years ago, none of this existed. AI systems were narrow — they did one thing, and one thing only. A chess engine played chess. A spam filter filtered spam. That was it. 

The real turning point came with the transformer architecture, introduced by Google researchers in a now-legendary 2017 paper called “Attention Is All You Need.”  

The transformer allowed models to process language in parallel rather than word by word, and to learn which parts of a sentence relate to which other parts — the attention mechanism we discussed in the previous chapter. This was the unlock. Suddenly, you could scale these things up, throw more data at them, add more computing power, and the models kept getting better. Nobody had quite expected that. 

Then came GPT-3 in 2020 — 175 billion parameters, trained by OpenAI on a chunk of the internet that would take a human thousands of years to read. It was the first time a model could write coherent essays, answer questions, and carry on a conversation at a level that genuinely surprised people outside the AI lab. The era of foundation models had begun. 

From there, the timeline reads like a science fiction novel on fast-forward. GPT-4 in 2023. Google’s Gemini 1.5 in early 2024. Meta open-sourcing LLaMA. And then, in January 2025, a Chinese startup called DeepSeek dropped a model that reportedly cost a fraction of what Western labs were spending — and matched them on benchmarks. The AI world had a very loud Monday morning. 

The Price Tag Nobody Talks About Loudly Enough

Here is where the story gets genuinely uncomfortable. Training a frontier foundation model is not like running a startup out of a garage. It is more like launching a space program. 

A single six-month training run for a model like GPT-5 is estimated to cost over $500 million in computing costs alone. GPT-4 is believed to have cost somewhere between $78 million and $100 million — and that was considered expensive at the time. Today, those numbers look almost quaint. Anthropic’s CEO has publicly said some models in development will cost over a billion dollars to train. 

And money is only part of the bill. Training GPT-3 consumed roughly 700,000 litres of water for cooling alone. A single large data centre can burn through 300 million gallons of water per year. OpenAI’s planned infrastructure buildout is expected to draw 2.2 gigawatts of power — enough electricity to power roughly one million homes. So the next time you ask an AI to write your weekly newsletter, know that it is, in a small but real way, drinking from somebody’s water table. 

The economics are dizzying even for the companies making money at it. OpenAI and Anthropic are growing revenues at unprecedented speed — we are talking billions annually. And yet they are simultaneously burning through capital at a pace that would make most investors nervous. The typical frontier model becomes commercially obsolete within three weeks of launch, because a competitor has already released something better or cheaper. It is a treadmill moving at rocket speed.