The Demo Passed. Who Signs? What a Buyer Should Demand From Agent-Built Software

An agent opens a pull request at 2 a.m. The pipeline runs. Tests pass, coverage holds, and the change merges before anyone in the building is awake.

By Thursday it is in production. By Friday, nobody can explain why it works — which is a different problem from it being broken.

That is the gap this analysis is about. Not whether an AI agent can write software; the evidence on that is settled. The open question is who signs for the result, and what makes a change signable.

The evidence position

Evidence for this edition was reviewed through 20:00 UTC on November 6, 2026.

Two limits belong at the top. The quantitative material below comes from a vendor-published survey, self-reported by its respondents and not independently audited. And no customer outcome, defect rate or savings figure is asserted anywhere in this article. The standards cited are guidance, and guidance is not a result.

What a demonstration actually proves

A demo proves a capability exists. It does not prove the capability is controlled.

Those are different claims, and the difference surfaces in the second month of production, when a rule the agent encoded needs to change and nobody can find where the rule lives.

Two numbers a buyer should hold together

CloudBees published a report it calls State of Code Abundance 2026, based on a survey of more than 200 enterprise technology leaders, and summarized the findings at a company event.

Two results sit next to each other. Nearly all respondents — 92% — expressed confidence in the production readiness of AI-generated code before it ships. And 81% said they had seen production issues increase, linked to that same code.

Both are true at once. The confidence is real, and so is the incident. CloudBees chief executive Anuj Kapur described the pattern at the same event as a missing scoreboard.

Three further figures matter to a buyer: 61% of the average enterprise codebase is now generated or assisted by AI, 70% say maintaining the test suite is a greater burden than writing code, and 12% have a dedicated AI governance team.

Attribute each one to a vendor survey rather than an audit. The direction is plausible; the sample is not large.

Why the scoreboard is blank

The measurement problem is specific. The same survey reports that 54% of organizations track AI productivity primarily by time saved — a number that describes how busy a team is, not what it delivered.

Fidelity Investments ran into the arithmetic. Gerard McMahon, who leads ALM tools and platforms there, described the pattern at that event: a productivity gain on the left side of the pull request did not produce the same gain on the right side, and senior engineers absorbed a disproportionate share of the review load.

That is a throughput problem wearing a productivity costume. Generation became cheap. Review, testing and deployment did not.

The review question has changed

Loreli Cadapan, a CloudBees product executive, put the shift precisely in that discussion: the review question is moving from "is this good code?" toward "do we actually understand what this code is doing?"

Code review was designed around an assumption — that the author could explain the change. An agent can generate a fluent explanation of code it wrote. That is not the same as an account of why it wrote it that way, and a reviewer cannot test the difference by asking.

Four artifacts that make a change signable

What follows is this publication's analysis, not a standard.

• The instruction. A specification stable enough that an engineer who was not present can state what the change was meant to accomplish.
• The failing test. A test that fails before the change and passes after. Without it, "tests pass" is a claim about the suite, not about the change.
• The named approver. A person with the authority to accept the risk, recorded against the diff — not a rubber stamp applied to a batch of forty.
• The rollback. A path back that has been demonstrated, in the environment where it would be needed.

None of these is exotic. All four are already routine in regulated delivery. What changed is that they can no longer be inferred from the fact that a human typed the code.

Where the agent's credentials live

The security question follows directly from the workflow. An agent that can only propose a diff is a drafting tool. An agent that runs tests, pushes branches and deploys is a principal holding credentials.

OWASP maintains a community list of the most critical security risks for applications built on large language models. Its categories include prompt injection and excessive agency — the failure mode where a system is granted permissions broader than the task requires.

Both describe a pipeline rather than a chatbot. An agent that reads a ticket can be instructed by that ticket. An agent holding a deploy token can deploy whatever an injected instruction asked for.

Boundary translation belongs in the same conversation. Microsoft's anti-corruption layer pattern describes a facade that translates between subsystems which do not share semantics, so that a dependency on an outside system does not limit the design of the one you control. Place that boundary between what a model proposes and what a system of record will accept, and the model stops being able to write directly into the ledger.

What the frameworks are actually for

NIST published its Secure Software Development Framework as SP 800-218, recommending a core set of high-level secure software development practices that can be integrated into a development life cycle.

The stated reason is the useful part for a buyer: because the framework provides a common vocabulary, software purchasers and consumers can use it to foster communications with suppliers in acquisition.

That is a narrow claim and a practical one. It hands you a list of questions a supplier already recognizes.

NIST is equally direct about the ceiling. It describes its Cybersecurity Framework as a tool to help organizations better understand and improve their management of cybersecurity risk. That is guidance, not a certification, and nothing in it certifies that a particular agent wrote a safe change.

Six questions for the vendor or the team

• For a merged change, which of the four artifacts exist, and where are they stored?
• What credentials does the agent hold, and what is the blast radius if one is misused?
• Can the agent approve its own work, or is verification independent by design?
• Does the team measure time saved, or throughput from pull request to production?
• Who owns an incident caused by a merged agent change, by name and by role?
• If the vendor's tool disappeared tomorrow, what would still run?

Three delivery options

Adopt the tool and keep the process. Cheapest and most common. It works where changes are small, reversible and reviewed by someone who understands the subsystem. It fails at the first change nobody can explain.

Build a control layer around delivery. Specifications, failing tests, independent approval and a demonstrated rollback, enforced where the merge happens rather than in a policy document. More work up front, and the only option that survives a change of tooling.

Govern what already shipped. Inventory what agents wrote, find the changes with no instruction and no named approver, and start there. Least glamorous, most likely to prevent the incident already forming.

Most organizations need the second and the third. The first is a decision about risk appetite, not about technology.

A 30-day acceptance pilot

• Days 1 to 5. Pick one delivery stream and one class of change. Not the portfolio.
• Days 6 to 12. For ten recently merged changes, try to produce instruction, failing test, named approver and rollback from records alone.
• Days 13 to 18. Score the gaps by consequence: which missing artifact would have let a bad change reach production unnoticed?
• Days 19 to 25. Enforce the single most consequential missing artifact at the merge gate, for that one stream, and nothing else.
• Days 26 to 30. Have someone who did not build the process reconstruct three changes from the record alone. If they cannot, the record is not finished.

Four indicators worth baselining

Merge-to-production time, measured from commit rather than from generation. Merged changes with no named approver, a number you intend to drive to zero. Review depth per change, however crudely you can count comments, rejections and revisions. Reconstructions that required asking a person, which is the honest measure of whether the records work.

Baseline all four before the pilot. None requires a vendor benchmark, and all four describe your operation rather than someone else's.

What is vendor-stated, and what is not

The CloudBees figures are a vendor's survey of its own market, presented at its own event and published on its own blog. They are useful signals about direction and unreliable as precision. No independent audit of that sample was reviewed for this edition.

The NIST, OWASP and Microsoft references are guidance documents, cited for the specific propositions described in the text. None is a certification, and none is presented as evidence that any organization has achieved a result.

Nothing here establishes that a control layer reduced incidents, shortened review time or improved delivery. The opening scenario is illustrative rather than a client story.

What remains unknown

• Whether the confidence-versus-incident gap in the survey holds at larger, independently drawn samples.
• How much of the reported review burden comes from agent-authored changes specifically rather than from volume generally.
• Whether any merge-gate control measurably reduces escaped defects, and at what cost in cycle time.
• How agent credentials are scoped in most enterprises today. No reliable inventory was reviewed.
• Any measured productivity effect, positive or negative. None is asserted.
• Market data. None was reviewed for this edition, so no stock movements are reported.

Evidence notes and limits

The survey figures — 92%, 81%, 61%, 70%, 54%, 12%, and the attributed remarks from Anuj Kapur, Gerard McMahon and Loreli Cadapan — come from CloudBees' own write-up of its State of Code Abundance 2026 report and its accompanying event. They are company-published and self-reported.

NIST SP 800-218 is cited for its described purpose of recommending secure software development practices and for its statement that the common vocabulary helps purchasers and suppliers communicate in acquisition. The NIST Cybersecurity Framework is cited only as a tool to help organizations understand and manage cybersecurity risk. Neither says anything about AI agents or code generation, and no control or certification claim is drawn from either.

The OWASP list is cited as a community-maintained catalog of LLM application security risks, and specifically for the existence of prompt injection and excessive agency as named categories. No specific finding or exploit is attributed to it.

Microsoft's anti-corruption layer pattern is cited for the described translation boundary between subsystems with different semantics. It is architectural guidance, not a measurement, and no effort, cost or outcome is attributed to implementing one.

The four artifacts, the criteria, the options, the pilot and the indicators are this publication's analysis and suggested method. They have not been validated as a protocol, and nothing here is a compliance requirement.

The advertisement above is separate from this analysis and is not a source for any claim in it.