The surge in AI workloads, particularly large language models (LLMs), has sparked a global wave of data centre construction and planning. Tech giants like Microsoft, Google, Amazon, Meta, and specialised providers such as NVIDIA and Equinix have earmarked tens of billions of dollars for new data centre investments in 2024–2025 alone. These modern data centres, designed for LLM hosting, are centred around GPU clusters and specialised accelerators. They feature advanced cooling methods (including liquid cooling), high power density, and new networking standards to meet the intense energy and bandwidth demands of language model training and inference.
Recently, data centre architects have begun incorporating sustainability KPIs to ensure technologies are deployed effectively and efficiently. In the 47th episode of my podcast, I spoke with Stargate project lead Rishab Vardhan Hariskishnan, who explained how data centres—as the foundation of AI infrastructure—interconnect with other critical infrastructure elements, such as energy, water, and fibre.
But what makes these modern AIs, especially LLMs, so demanding in terms of computational resources?
The Stargate project, for instance, is a $500 billion joint venture between Oracle, OpenAI, SoftBank, and MGX. Unveiled in January 2025, Stargate aims to satisfy the surging demand for AI processing power by implementing 10 gigawatts of advanced data centre capacity, driven by millions of GPUs.
Let’s examine what occurs when ChatGPT and similar models generate a ‘response.’
Mainstream large language models (LLMs) like ChatGPT rely on running large matrix multiplications across all layers, connections, and weights for every token processed. This results in high GPU energy demand, even if only a small part of the model’s knowledge is relevant for a given text.
What does it mean in other words?
When you query an LLM, producing each token involves a forward pass through all layers of the neural network. This process involves multiplying massive matrices (constructed from billions of weights) with input data—a task GPUs are well-suited for, given their ability to perform rapid, high-volume parallel computations. During token generation (inference), all layers and most connections are computed for the current input and ongoing context. Essentially, the entire model runs through every layer each time—a process known as a forward pass—so the GPU performs calculations using all weights for each token generated.
With each step, the complete array of weights and connections must be loaded into GPU memory and processed, along with the activations, which serve as temporary storage for the model’s current state as it works through tokens one by one. When working with large language models (LLMs), we are examining the full model activations at any given moment. Aside from some optimised or quantised models, this computation occurs for every single token during text generation, not just during the initial setup phase. Therefore, if you seek a lengthy response, the model performs all these calculations anew for each additional word it proposes. Most models do not offer any shortcuts.
Developers of LLMs strive to enhance their technologies for improved energy efficiency.
One method is called Mixure of Experts. In this approach, only a subset of model parameters (“experts”) is activated for each token, dramatically reducing the compute required per token and making inference more energy-efficient. Google, Microsoft, Meta, and others use MoE in their large models—for many tasks, only the most relevant “experts” are activated.
Additionally, optimised architectures like linear RNNs (DeltaNet, Mamba, xLSTM), spiking neural networks, and memory-efficient variants are employed. These approaches circumvent quadratic computations or utilise less energy-consuming mathematics.
However, none of these methods will create an AI-energy ecosystem that operates like biological systems.
Neurosymbolic and Hybrid models, which integrate neural networks with clear symbolic reasoning and logic—such as ontologies, rules, and knowledge graphs—are still in their early stages, particularly when it comes to causal representation. This involves learning representations that distinctly differentiate between causes and effects or breaking knowledge into separate, manageable “causal variables,” allowing for exploration of hypothetical scenarios. Furthermore, language models developed using graph structures, rather than just sequential data, are still nascent and underfunded.
The limitations of large language models (LLMs), despite the vast amounts of money being invested in data centres, may eventually have repercussions on the AI bubble and could potentially lead to its collapse.