The Model Is Rented. The Evidence Has to Be Yours: What Embedded AI Owes an Application

An AI feature can change behavior without a deployment, and most acceptance tests measure a snapshot of someone else's model rather than a property of your system. This analysis takes NIST's published cautions seriously: capability is difficult to predict, results on one task do not reliably transfer to another, and misuse risk is not a function of the model alone. It maps the three evaluation levels used in NIST's ARIA program, the change-management and incident records the Generative AI Profile asks for, and the human-override data the AI Risk Management Framework says is worth collecting. Three build options, four questions, and what remains unestablished. No customer result, benchmark or answer-quality figure is asserted.

An AI feature is the only line item in a business application that can change its behavior between deployments. That is the whole problem in one sentence.

You ship grounded answers. They pass acceptance testing. Six weeks later the tone is subtly different, one retrieval path returns stale text, and nothing in the release notes explains either.

That is not a model going rogue. It is a record that was never kept.

What this analysis can and cannot support

Three limits belong at the top.

First, everything below about a specific product is the vendor's own published description, labeled as such, and is not independently tested here. No answer-quality benchmark, latency figure, accuracy rate or customer outcome is asserted anywhere in this article.

Second, the NIST material cited is guidance. The Institute describes these documents as voluntary and non-certifying, and none of them establishes that any particular embedding is safe, accurate or well governed.

Third, the questions and options are this publication's analysis. They have not been validated as a protocol.

The capability was never the constraint

NIST's second public draft of AI 800-1, published January 2025, opens its challenges section with a line that reads like it was written for procurement committees.

Misuse risks, it states, "are not a function of the model alone—they result in part from malicious actors' motivations, resources, and constraints, as well as the ways that models are integrated into applications and society's defensive measures against harms."

The same document is candid about what scale tells you. Increasing training data, compute or parameters can improve performance, but those factors are "useful heuristics in some instances" with "limited precision and an uncertain relationship to a model's capabilities and potential misuse risks."

Translated for a product team: the capability you evaluated is not the property you are depending on. The integration is.

Your acceptance test measured a snapshot

The sharper warning is about how evidence travels between tasks.

In AI 800-1, NIST writes that model performance on one task "may not provide reliable evidence of its performance on others, even when the two tasks appear related."

Read that against a typical grounded-answer acceptance test. Twenty questions, a rubric, a sign-off. That test establishes that the system behaved on twenty questions on one afternoon.

It does not establish a property of your application. It establishes a reading of a rented model, and rented models are re-read constantly.

Three levels of evaluation, and most teams stop at the first

NIST's Human-Centered AI program describes how its ARIA work assesses AI applications, and the structure is more useful than most internal test plans.

Three levels: model testing to confirm claimed model capabilities and the performance of model guardrails; red teaming to stress-test applications and guardrails and elicit disallowed behavior; and field testing to test the application in regular use.

Most enterprise AI programs complete the first, document it once, and consider the matter closed. The third is where the answers your users actually see get measured, and it is the one most likely to be cut when a launch date moves.

The Generative AI Profile is equally direct about the ceiling of pre-deployment work. Anecdotal testing of capabilities through games or standardized tests designed for humans, it notes, "does not guarantee GAI system validity or reliability in those domains." Nor does it treat prompt engineering as a substitute for assessment.

The record is the part you own

Here is the practical asymmetry. You cannot control how a vendor's model is retrained. You can control whether your application leaves a trail.

The Generative AI Profile's manage actions are specific enough to implement. One calls for change-management records and procedures for GAI systems — the reasons for each change, how it could affect each intended context of use, and step-by-step detail of how changes were planned, tested and deployed.

Another asks for policies and procedures to record and track reported errors, near-misses, incidents and negative impacts. A third calls for after-action assessments of GAI incidents. A fourth calls for regular sharing of information about errors and their severity.

None of that is exotic. It is the discipline most software organizations already apply to production incidents, applied to a component that was quietly exempted from it.

Human override is data, not sentiment

One line in the AI Risk Management Framework's appendix on human-AI interaction deserves a place on a product roadmap.

Data about "the frequency and rationale with which humans overrule AI system output in deployed systems," the appendix states, "may be useful to collect and analyze."

The same appendix warns that oversight roles need to be clearly defined and differentiated, and that under some conditions the AI part of a human-AI interaction can amplify human bias toward more biased decisions than either the human or the AI alone.

So override is not a courtesy feature for skeptical users. It is your cheapest continuous evaluation signal, and it only exists if the override is recorded with a reason attached.

Three places an embedded assistant can live

Every embedded AI request resolves to one of three architectures, and the choice is a budget decision before it is a technical one.

Call the model directly from the feature. Fastest to ship. The application holds no context, no evaluation set and no record of what was asked, which means the next regression is undiscoverable.

Build a thin orchestration layer you own. Prompts, retrieval, evaluation cases and override logs live in your codebase, calling models behind an interface you control. More work up front, and the only option where a model change becomes a diffable event.

Adopt a platform that carries context and research. The published description of Bordy, cited below, distinguishes application-level intelligence — research, context and memory inside an application — from developer-facing intelligence covering coding, QA and orchestration, with capabilities and entitlements confirmed per integration.

Most organizations end up blending the second and third. The failure mode is not choosing wrong once. It is choosing direct calls by default, repeatedly, until nobody can say which features depend on which model.

Four questions before you ship an embedded feature

What did we test, and on which date? If the acceptance evidence is twenty questions from one afternoon, write that down honestly. A snapshot labeled as a snapshot is manageable. A snapshot presented as a property is a future incident.
Where does a model change show up? In a release note, in a vendor changelog that nobody reads, or nowhere. If the answer is nowhere, the feature is not yet operable.
Who owns the prompt and the retrieval set? A named person, not a team. Prompts and corpora that live in a vendor console are assets you cannot diff.
Do we capture overrides with reasons? If not, you are discarding the only continuous evaluation signal your users generate for free.

What remains unestablished

Whether any specific embedded assistant improves task completion, answer accuracy or user satisfaction at any organization. Nothing here establishes an outcome.
What Bordy's capabilities are for any particular environment. The published descriptions distinguish application-facing and developer-facing intelligence and state that capabilities, tools, access and entitlements are confirmed per integration.
Whether the NIST manage actions, applied to an embedded assistant, reduce incidents. They are recommended practices, not a tested control set.
Which architecture performs better at any given scale. That is a per-estate question and no cost or effort figure is asserted.
Market data. None was reviewed for this edition, so no stock movements are reported.

Evidence notes and limits

NIST AI 800-1, Managing Misuse Risk for Dual-Use Foundation Models, second public draft, January 2025, is cited for its statements that misuse risk is not a function of the model alone, that scale factors carry limited precision and an uncertain relationship to capability, and that performance on one task may not provide reliable evidence of performance on another. It is a voluntary guideline document, explicitly not a certification.

NIST's Generative AI Profile, AI 600-1, is cited in its initial public draft for the manage actions on change-management records, error and incident tracking, after-action assessments and information sharing, and for its caution that anecdotal testing does not guarantee validity or reliability. The specific action identifiers are those printed in the document.

The NIST Human-Centered AI program page, updated June 24, 2026, is cited for the three-level ARIA assessment structure: model testing, red teaming and field testing. It describes a program, not a requirement.

The AI Risk Management Framework appendix on human-AI interaction is cited for its statement that data about the frequency and rationale of human overrides may be useful to collect and analyze, and for its cautions about defining oversight roles and about AI amplifying human bias in some conditions.

Statements about Bordy are the company's own published product descriptions: chat with optional in-depth research, selective personal memory, conversation transfer, shared rooms and channels, and image analysis; a distinction between application-facing and developer-facing intelligence; per-integration confirmation of capabilities and entitlements; and the two published trial terms. They are vendor claims, not independently verified results, and no customer outcome is attributed to them.

The three architectures, the four questions and the reading of the NIST material are this publication's analysis and suggestion. Nothing here establishes that any organization improved accuracy, reduced incidents or recovered staff capacity.

The advertisement above is separate from this analysis and is not a source for any claim in it.