testing

The solution is to write tests for features in such a way that they are independent of the code. I like to use the neural network test for this: Can you re-use the test suite if your entire software is replaced with an opaque neural network?

A good test suite is, first and foremost, a risk-mitigation measure.
Automated tests reduce the risk associated with changes to an existing codebase - most regressions and bugs are caught in the continuous integration pipeline and never reach users. The team is therefore empowered to iterate faster and release more often.

Pure test do little-to-no IO, they are independent of timings and environment. Less pure tests do more of the impure things. Purity is correlated with performance, repeatability and stability. Test purity is non-binary, but it is mostly discrete. Threads, time, file-system, network, processes are the notches to think about.

Don’t think about tests in terms of opposition between unit and integration, whatever that means. Instead,
Think in terms of test’s purity and extent.
Purity corresponds to the amount of generalized IO the test is doing and is correlated with desirable metrics, namely performance and resilience.
Extent corresponds to the amount of code the test exercises. Extent somewhat correlates with impurity, but generally does not directly affect performance.

Architecture the software to keep as much as possible sans io. Let the caller do input and output, and let the callee do compute. It doesn’t matter if the callee is large and complex. Even if it is the whole compiler, testing is fast and easy as long as no IO is involved.

Ruthlessly optimize purity, moving one step down on the ladder of impurity gives huge impact.
Generally, just let the tests have their natural extent. Extent isn’t worth optimizing by itself, but it can tell you something about your application’s architecture.

LLMs are not great at following commands. They deviate. Frequently. And they’re unreliable at self-verification—they’ll confidently tell you the code works while it’s on fire. The fix isn’t to ask the LLM to verify. It’s to ask it to write a script that verifies. Shift from judgment to artifact.

Instead of asking an LLM “Did this work?” you define verification steps that produce a series of pass/fail artifacts. The agent can’t negotiate with a failing test. It either meets the specification or it doesn’t.

Delete Cargo Integration Tests

My solution to this problem is making the tests data driven. Instead of every test interacting with the API directly, I like to define a single check function which calls the API.

An agent optimizing for passing tests will find ways to pass them. If the tests were written after the implementation, they are likely testing what the implementation happens to do rather than what it should do.

The infrastructure that needs to exist to support this model at scale includes better automated regression detection, artifact-level validation that goes beyond diffing changed lines, reliable and fast environment provisioning, and guardrails that hold up under parallel workloads.