Part 1 of a series on what’s broken in the EU AI Act — and why it matters
Let me be direct with you: the EU AI Act is a historic achievement — and a deeply flawed one.
It took years of political will, thousands of pages of negotiation, and genuine ambition to protect citizens from AI harms. I respect that effort. But somewhere between the first draft and the final text, something went wrong. The Act was written by people who understand products, liability frameworks, and regulatory checklists — but who fundamentally misunderstood what AI systems actually are, how they behave, and where the real dangers lie.
The result? Europe now carries a €3.3 billion annual compliance burden — and the most dangerous AI risks remain structurally unaddressed.
That’s not governance. That’s the governance illusion.
The Numbers Don’t Lie
Before diving into the technical failures, let’s talk economics — because this matters enormously for Europe’s position in the global AI race.
DIGITALEUROPE estimates that EU AI Act compliance will cost €3.3 billion per year across the economy. For a 50-person SME building a high-risk AI tool, that’s €320,000–€600,000 in upfront compliance costs, plus up to €150,000 every year after that. These are not numbers that a startup in Munich or Warsaw can easily absorb — especially when their competitors in San Francisco face no equivalent burden.
The Draghi report didn’t mince words either: over 60% of EU companies see regulation as an obstacle to investment. Meanwhile, Apple and Meta delayed AI feature launches in Europe. Meta publicly refused to sign the GPAI Code of Practice, calling it legally ambiguous. As of mid-2025, only three Member States had even designated the national competent authorities the Act required by August 2025.
This is not the portrait of a regulation that is working.
Failure #1 — What Even Is an “AI System”?
Here’s where things get philosophically interesting — and practically dangerous.
Article 3(1) of the EU AI Act defines an AI system as: “a machine-based system designed to operate with varying levels of autonomy… that infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions.”
Sounds reasonable, right? Except legal scholar Philipp Hacker from the European University Viadrina put his finger on a fatal flaw: “In this unqualified form, this is a definition of software, not of AI. Take an auto-sum function in an Excel sheet. It has an objective, input, and an output. The only distinguishing criterion on this definition is ‘infers’.”
The entire weight of separating ordinary software from regulated AI systems now rests on a single technical word — one that legal document authors, not AI engineers, will interpret in practice. That is an enormous loophole dressed up as precision.
And it gets more complex. When a large language model is fine-tuned and embedded in an agentic pipeline performing a task listed as “high-risk” in Annex III – is it the original model? The fine-tuned variant? A new system entirely? As of today, that question has no settled answer.
Failure #2 — You Can’t Audit What You Can’t Track
Here’s a scenario I want you to sit with for a moment.
Imagine a multi-agent AI pipeline at a bank: Agent A retrieves customer data. Agent B reasons over it. Agent C executes a loan decision. Something goes wrong — a discriminatory outcome, a hallucination, a security breach. Who is responsible? Which agent caused the harm? When exactly did it happen?
Under the EU AI Act, you cannot answer those questions. Why? Because the Act operates at the product registration level – it requires manufacturers to register their AI systems, not to attach persistent identifiers to individual agent instances at runtime. There is no equivalent of a session token or UUID for agentic workflows anywhere in the Act or the GPAI Code of Practice.
The Act addresses the manufacturer’s front door. It completely ignores the assembly line.
This is not a minor technical quibble. Multi-agent pipelines are being built and deployed right now by every major company. Without agent-level traceability, post-incident audits are structurally impossible — not difficult, not expensive, but impossible.