Do you remember the scene in Fantasia where Mickey Mouse enchants a broom to do his chores?
It works brilliantly. Then the broom multiplies. Soon an army of enchanted brooms is performing the task with astonishing productivity while flooding the castle.
Mickey’s problem is not that the spell failed. It worked. He simply does not know the stop command.
Software engineering feels a little like that right now.
We handed AI the magic hat, and it is happily producing thousands of lines of code before we have finished our morning coffee.
At least, that is increasingly how I work.
First thing in the morning, I review the tickets, fire off an agent and get coffee. By the time I return, the code is waiting.
Bang. Done for the day.
Of course not.
There are still four hours of meetings, several new bugs, a build failure nobody caused and a “quick question” that develops its own roadmap.
But the code-generation part can feel almost absurdly fast.
The tests are green. The sprint board is clearing. Developers are collecting story points like arcade tickets.
Meanwhile, the water is rising in production, bugs are swimming around the server racks, and we are looking at one another asking:
Does anybody here actually know how to read this spell?
Everyone can see that code is arriving faster than it can be understood.
Nobody wants to be the first person to slow down and grab a mop.
That is the part of AI coding that worries me most.
It is not only a software-quality problem.
It is a game-theory problem.
The tool changed the competition
For most of software history, implementation was expensive.
Organizations had more ideas than developers could turn into code. Writing the software was the obvious bottleneck.
AI weakens that constraint.
It can create functions, tests, infrastructure and entire applications before the requirements meeting has finished deciding whether the button should say Continue or Next.
The spell really does work.
The productivity evidence is mixed. A GitHub study found benefits on a bounded API task, while METR found that experienced open-source developers using early-2025 tools took longer in repositories they knew; its 2026 update suggests newer tools probably help more, though the size is hard to estimate. DORA describes AI as an amplifier of the organization adopting it. (GitHub, 2024; METR, 2025 and 2026; DORA, 2025)
We do not know the universal productivity multiplier.
We do know what organizations believe they are buying.
More output.
And once more output becomes possible, producing less requires an explanation.
Game one: inside the organization
Imagine two developers.
One uses AI to finish ten visible tasks. The other finishes four, rejects an unnecessary feature, divides a large change into reviewable pieces and tests failure cases.
The first has ten completed tickets. The second has six incidents that never happened.
Unfortunately, the performance dashboard has no column for those.
There is no Jira ticket for:
Prevented an outage nobody will ever know about.
There is only a lower feature count.
So even if both developers understand the risk, their incentives are different.
The coordination trap
This is related to the problem in why incentives produce the opposite behaviour.
If management rewards tickets closed, pull requests merged or features shipped, developers will use AI to improve those numbers.
That is not gaming the system.
That is following the system’s instructions with admirable precision.
Game two: company against company
Now move one level up.
Suppose two companies know AI-generated change needs more verification. Both would prefer careful adoption, but one sees its competitor releasing features every week. Customers see the features. Investors see the demos. The incident avoided by careful review remains invisible.
What is the CEO supposed to announce?
We are proud to report that our competitor shipped twelve features, while we deeply understood four.
That may be the responsible decision.
It is not an easy slide for the quarterly presentation.
So each company accelerates because slowing down alone may mean losing customers, talent, funding or the story that it is “leading in AI.”
The result is an arms race nobody needs to coordinate explicitly.
Each decision can be rational.
Together, they can produce an industry creating software faster than it can understand the consequences.
The assurance gap is the scoreboard we do not show
I use assurance gap to mean the difference between:
how quickly an organization can produce software changes
and:
how quickly it can understand, test and safely operate those changes.
The gap does not require terrible code. There is simply more of it arriving at review, security, testing and operations.
Those stages depend on scarce human attention and knowledge of context. A study of Chromium OS security reviews found that vulnerabilities spanning broader context and more directories were more likely to escape review. (Paul, Turzo & Bosu, 2021)
Before AI, some of the time spent writing code was also time spent learning the system.
You discovered why the “simple” service had three queues, two clocks and a database field named temporary_final_status.
When AI removes the friction, it may also remove part of that accidental education.
The organization gains behaviour faster than its people gain understanding.
That is comprehension debt.
It is the waterline rising while the board continues reporting that broom utilization has never been higher.
The bill arrives during an incident, usually after somebody asks in Slack:
Who knows this service?
Several people begin typing.
Then stop.
Can the AI review the AI?
Of course—and it should.
AI can generate tests, inspect changes, search for vulnerabilities and explain unfamiliar code. These tools may improve assurance as quickly as generation—but the coordination problem remains.
If AI review makes a team capable of safely absorbing twice as much change, the organization may respond by asking for three times as much change.
The new capacity becomes a new target.
And if the generator and reviewer share similar assumptions, we may occasionally get the same confidence twice rather than two independent checks.
An AI reviewer telling an AI author that the implementation is “well-structured and comprehensive” is useful evidence.
It is not quite the separation of powers.
Change the payoff, not the speech
Telling developers to “use AI responsibly” will not solve a game in which visible acceleration is rewarded.
The organization has to make restraint something other than a unilateral career sacrifice.
That could mean:
- treating review and operational capacity as real limits on feature production;
- keeping generated changes small enough for an accountable person to understand;
- attaching testing, observability and rollback work to the feature rather than to a later “hardening” phase;
- rewarding deletion, rejected complexity and incidents prevented—not only code added;
- and naming a human owner who can explain consequential changes without opening the chat history.
These practices can become new metrics to game. People are creative, especially when a promotion cycle is approaching.
But the goal is not a perfect scoreboard.
It is to stop pretending that generated output and verified progress are the same thing.
The dangerous move may be the rational one
No developer wants to become the person who needs three days to review what a colleague generated before lunch.
No engineering leader wants to tell the board that the company is adopting the future at a measured pace.
No company wants to discover that caution was admirable immediately after losing the market.
So everyone accelerates.
The deepest risk may not be that AI writes uniquely bad code.
It may be that AI makes producing unverified change so individually rewarding that nobody can afford to stop producing it.
The scarce resource is no longer code.
It is the attention required to know what the code has made true.
AI can conjure more code than ever before.
But when the castle floods, who gets to slow down and grab a mop?
Until the next strange question,
Osagie