generated-code

I knew we had a problem about six months ago when a member of our sales organization reached out to me asking for API keys. Keys plural. They used AI to build what they described as a SuperApp that would transform our go-to-market teams. All they needed was production access to about a dozen systems of record at Cloudflare and admin permissions to a deployment pipeline to make it work.

But here is the thing: if you showed a video generation model enough footage of people using spreadsheets, it would produce the exact same result. You type =A5+1, and a 2 appears. Not because anything was computed. Because the model knows what happens next. It has seen this scene ten thousand times. The 2 is not the output of arithmetic. It is the output of pattern recognition so thorough that it becomes indistinguishable from arithmetic. I mean, nothing really crazy if you think about it. It’s the same principle that allows ChatGPT to write you a perfectly functional Python script despite having never, in any meaningful sense, learned to program. It just knows what code looks like when it’s going well.

Standards will catch up. I’ve been out of the web standards space for a couple years now, so this is pure speculation on my part. But I imagine that a lot of the efforts to improve the ergonomics of building websites – better CSS shorthands, terser JavaScript syntax, etc. – will become perceived as less important relative to things that actually move the needle on performance, capabilities, etc. At the end of the day, it’s just not very different for an agent to write 3 lines of CSS instead of 1, and anyway using the newer syntax might actually be harder because you have to coach the agent about things that aren’t in its training weights.

I no longer have a firm mental model of what they can do and how they work, which means each additional feature becomes harder to reason about, eventually leading me to lose the ability to make confident decisions about where to go next.

And so generating code is not necessarily the part that you need to worry about the most. Given enough context, agents can write the implementation and run the tests and inspect failure and revise code for us. We need to get to a place where we feel like there is enough of human taste encoded in the environment that we can trust what is being built, so that our human attention can be focused on the places where it’s needed most.

Comprehension debt is the growing gap between how much code exists in your system and how much of it any human being genuinely understands.

Dr. Margaret-Anne Storey, professor of computer science at the University of Victoria, first used the term ‘cognitive debt’ in October 2025 while teaching an entrepreneurship startup course. She had encouraged her students to use AI to move faster, and that worked well: the students were putting products into the hands of users and getting feedback. However, they were struggling with implementing that feedback back into their product.

Cognitive debt, a term gaining traction recently, instead communicates the notion that the debt compounded from going fast lives in the brains of the developers and affects their lived experiences and abilities to “go fast” or to make changes. Even if AI agents produce code that could be easy to understand, the humans involved may have simply lost the plot and may not understand what the program is supposed to do, how their intentions were implemented, or how to possibly change it.

We argue that coding agents have crossed a threshold of capability at which traditional human code review is no longer a necessary component of a software quality pipeline. Our argument rests on two claims: every stated goal of code review can be served by agents at lower cost and higher throughput; the naive integration in which agents write code and humans remain the mandatory reviewers is a dead end because it neither provides meaningful assurance nor scales with AI-assisted throughput.

We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.

Drawing on telemetry from over 10,000 developers across 1,255 teams, Faros’ recent landmark research report confirms:
Developers using AI are writing more code and completing more tasks
Developers using AI are parallelizing more workstreams
AI-augmented code is getting bigger and buggier, and shifting the bottleneck to review
Any correlation between AI adoption and key performance metrics evaporates at the company level

Extensive qualitative analysis of enterprise software engineers reveals that AI’s impact on the SDLC is not a simple linear improvement. Instead, it presents a series of profound tradeoffs. While AI successfully accelerates initial code generation and reduces the friction of starting new tasks, the time saved in creation is frequently re-allocated to auditing and verification. This tension may explain some of our own findings: higher AI adoption is associated with an increase in both software delivery throughput and software delivery instability.

The 2026 GenAI Code Security Report found that roughly 44% of AI code generation tasks introduced a risky security vulnerability in tests. The average security pass rate across models is 56% – barely changed from 55% in the first report. In other words, security performance has stayed flat while the amount of AI-generated code entering pipelines has surged.
That is why GenAI code security is now a scale problem, not a theoretical risk discussion. If AI is responsible for half the codebase and vulnerable output remains this common, every organization needs a sharper strategy for model selection, verification, remediation, and governance.

Reviewing AI generated code can be tough, especially if the code generated is solving a particularly complex problem. Even if the code is correct, it can be hard to understand how exactly it works. When working with other human programmers, I would generally ask them to walk me through their code, and I would ask questions about why they made certain decisions as we go. I wanted a similar process for reviewing AI generated code. After a lot of exploring, I've found a combination that works well for me.

AI-assisted code authorship has continued its rapid ascent, and the most motivated developers are shipping more code than ever. But four years of code-change data suggest maintainability signals sliding backward: cross-file function calls (indicative of reuse) are down 35%. Refactoring line moves are down 70%, and long-term legacy maintenance is down 74% vs 2022 levels. Concurrently, we observe a concerning rise in within-commit copy/paste (+41%), code block duplication (+81%), error-masking constructs (+47%), and two-week code churn (+15%). The throughput is real, but so is the debt it accrues, and that debt concentrates among developers who haven’t recognized the failure modes that endanger long-term repo maintainability.

Building the infrastructure to make verification routine is now a national-scale engineering problem.