AI Needs Guardrails: Gaining the Benefits Without Losing Control

Executive summary

Most enterprise AI programmes are still governed as if the technology were a text tool. A chatbot that drafts a reply is a text tool. An agent that issues the refund, updates the customer record and emails the customer is an operating component of the business, and it warrants the same controls any other component with write access would receive.

That distinction — authority rather than intelligence — organises everything that follows. Guardrails are best understood not as a content filter but as the conditions under which delegation becomes rational: what the system may read, what it may change, whose authority it carries, who can stop it, and what evidence remains afterwards.

Three points frame the analysis. First, guardrails are a system property distributed across five surfaces, and controls that work on one routinely fail on another. Second, the controls that hold under pressure are enforced by code the model cannot negotiate with; a safety instruction is not one of them. Third, voluntary frameworks such as NIST's are governance structures, not compliance regimes, so legal obligations arising from sector regulation, contract and data-protection law must be assessed separately.

The line that matters is authority, not intelligence

Two systems can share a model, a prompt template and a user interface and still carry completely different risk. In the first, an assistant drafts a refund response for a support agent to approve. In the second, an agent issues the refund, updates the record and notifies the customer. The difference is not capability. It is permission to act.

Business cases rarely make this explicit. Procurement discussions tend to focus on accuracy, latency and cost per seat, because those are the metrics a demonstration can display. Consequence is harder to display and far more expensive to get wrong. A wrong answer costs a review. An unauthorised action costs a customer, a reconciliation, a filing or a disclosure obligation.

The practical implication is that the same platform may be entirely appropriate for one use and inappropriate for another without any change to the underlying model. Governance has to attach to the specific workflow and the specific consequence, not to the vendor relationship.

Five control surfaces

Data. What the system may retrieve, at what freshness, under which classification rules, and with what restrictions on cross-border or cross-entity access. Retrieval scope is a records-management decision before it is an AI decision.

Identity. Whether the AI acts as itself, as the requesting user, or as a narrowly scoped service identity — and how quickly any of those can be revoked. An agent inheriting a departing employee's standing permissions is an access-control problem wearing a new label.

Model behaviour. The reliability of reasoning, the calibration of confidence, the handling of ambiguity, and the tendency to produce fluent output that is simply wrong.

Tool access. Which functions the system can invoke, with what parameters, under what rate and spend ceilings, and how results are validated before they are trusted downstream.

Operational governance. Approval thresholds, escalation rules, audit durability, evaluation cadence, incident response and the authority to suspend the system.

Failures cluster at the seams. A correct answer produced under the wrong identity. A valid action taken against another tenant's record. A permitted tool invoked in an impermissible sequence. Reviewing each surface in isolation will not surface these, which is why the design conversation has to be cross-functional.

What actually goes wrong

Prompt injection arrives through ordinary business content

Any text the system ingests — an inbound email, a supplier PDF, a web page, a support ticket, a counterparty's document — can contain instructions. If the agent treats retrieved content as instruction rather than as data, an outside party can influence its behaviour without ever authenticating. This is the most underappreciated exposure in agentic deployment precisely because it arrives through channels the organisation already treats as routine, and because no breach of the model is required.

Permissions, tenancy and the retrieval problem

Retrieval-augmented systems assemble context at runtime. If retrieval scope is governed by a written instruction rather than by the authorisation model, a user can receive a synthesis of records they were never entitled to see, presented in the form of an ordinary answer. In a multi-tenant platform, isolation enforced in prompt text is not isolation at all; if the data layer does not enforce tenancy independently of what the model was told, a retrieval defect becomes a cross-customer disclosure. Broad credentials, granted to make a pilot useful quickly, are the weakness that most often survives into production.

Reliability across steps, and cascading error

Single-turn accuracy is a poor predictor of multi-step reliability. An agent that plans correctly at step one but misreads system state at step four can complete an entire workflow in the wrong direction with high apparent confidence. Because automated systems consume each other's output, one plausible but incorrect record can propagate through reconciliation, reporting and downstream decisions before a human notices anything anomalous.

Accountability and cost

When an agent acts, responsibility is frequently unclear until an incident forces the question: the operator, the model provider, the workflow owner or the approving manager. Cost behaves similarly. Loops, retries and recursive tool calls consume tokens and API budget, and without step budgets and spend ceilings a malfunctioning agent can be expensive well before it is visible.

Why a safety prompt is not a control

A system prompt is an input to a probabilistic component. It can shape behaviour and it has legitimate uses, but it is not an enforcement boundary, because it competes with every other token in the context window — including content the organisation does not control. Instructions such as 'never access another customer's data' or 'do not send email without approval' are requests to a model, not constraints on a system.

Controls that survive contact with production share one property: they are enforced by code the model cannot negotiate with. Server-side authorisation checks. Row-level tenancy in the database. Tool allowlists. Approval gates in the workflow engine. Spend caps in the orchestration layer. The model may propose; the platform disposes.

What the guidance says, and what it does not require

NIST released the Artificial Intelligence Risk Management Framework (AI RMF 1.0) on 26 January 2023, published as NIST AI 100-1. NIST describes it as a resource for organisations designing, developing, deploying or using AI systems, and states that the framework is intended for voluntary use. The document is deliberately non-sector-specific and use-case-agnostic, an approach intended to let organisations of different sizes and in different industries apply it according to their own risk profile.

Structurally, the framework organises activity around four core functions — govern, map, measure and manage — supported by a companion Playbook and a set of crosswalks to other standards. NIST has also published guidance addressing risks specific to generative systems, and the framework itself is written to adapt as the technology develops rather than to freeze practice at a point in time.

Three status points matter for executives making commitments now.

• It is voluntary. The framework is a reference structure for governance, not a compliance regime, and it does not by itself create legal obligations.
• It is not a checklist. NIST presents trustworthiness as a set of characteristics — including validity and reliability, safety, security and resilience, accountability and transparency, explainability, privacy enhancement, and fairness with harmful bias managed — that interact in practice. Controls that improve one can degrade another.
• Version currency is a live question. Because the framework is designed to adapt, teams should confirm the current edition and any companion guidance directly rather than treating a pinned copy as final.

On the application-security side, OWASP publishes a widely used taxonomy of risks in LLM-based applications, including prompt injection and excessive agency, which many security teams use as a working reference for control design. The project is revised periodically; readers should consult the current edition directly for its exact scope, and practitioners should treat it as a taxonomy for building controls rather than a certification standard.

Separately, legal obligation does not disappear because a framework is voluntary. Sector regulators, contract terms, data-protection law and consumer-protection rules all continue to apply to automated decisions, disclosures and records. The governance question is not 'are we aligned to a framework' but 'can we evidence that this specific use, with this specific consequence if wrong, is controlled and accountable'.

Advisory assistant versus acting agent

A decision framework: five gates before deployment

Gate one — name the decision, not the capability. Specify the workflow, the business object and the measurable baseline before selecting a model. A capability statement cannot be evaluated; a workflow can.

Gate two — classify consequence. Separate read-only, reversible and irreversible actions. Controls should scale with consequence, not with enthusiasm for the technology.

Gate three — map the data path. Document every source the system can reach and the authorisation rule that governs each. If the map cannot be produced, the deployment is not ready for review.

Gate four — fix the identity model. Decide whether the agent acts as the user, as a scoped service identity, or both. Make revocation routine rather than exceptional, including for every tool credential the agent holds.

Gate five — prove reconstructability. Confirm the platform can produce a durable record of the plan, the retrieved sources, the tools invoked, the approvals granted and the outcome. A chat transcript is not an audit trail.

Implementation steps that hold up

• Least privilege with server-side authorisation. Effective permissions should be the intersection of what the task requires and what the requesting user is entitled to, enforced in the API and data layer.
• Tenant boundaries at the storage layer. Isolation evaluated on the query path, independently of model output.
• Approved tool allowlists. A defined set of callable functions with typed parameters and documented side effects; anything not listed is unavailable by construction.
• Sandboxing. Code and file operations executed in isolated environments with no ambient credentials and no default network reach.
• Human approval for consequential actions. Sending external communications, moving money, deleting records or altering a legal obligation should require a human decision unless explicitly authorised otherwise.
• Bounded autonomy. Step limits, recursion limits, spend caps, rate limits and hard stops. Autonomy should be a dial, not a switch.
• Validation at every boundary. Schema validation on tool parameters, policy checks before execution, and post-condition verification that the intended state change actually occurred.
• Evaluation and red teaming. Test against adversarial content, injected instructions and ambiguous requests before deployment and after every material change. Measure completed work against system state, not the plausibility of the prose.
• Monitoring and incident response. Alert on anomalous tool use, permission denials, unusual retrieval patterns and cost spikes. Rehearse the suspension procedure; an untested stop control is a documented intention, not a safeguard.

The tradeoffs, stated plainly

Guardrails cost something, and pretending otherwise undermines the case for them. Approval gates add latency and can produce rubber-stamping if thresholds are set carelessly. Narrow permissions mean more integration work. Sandboxing complicates tooling. Evaluation programmes consume engineering time that stakeholders would rather spend on features. Alerting generates false positives, and enough false positives produce the alert fatigue that precedes an unnoticed incident.

The honest framing is that these are the operating costs of delegation. An organisation unwilling to fund them is not choosing speed; it is choosing to discover its control gaps through incidents rather than through design. The countervailing point is equally real: controls applied indiscriminately to low-consequence work make the programme slow enough that teams route around it, which is a worse outcome than a proportionate control.

Measuring benefit honestly

Value claims for AI are frequently asserted at the level of the platform rather than the process. A more defensible approach is to fix a small number of operational measures before deployment and read them afterwards: cycle time for the specific workflow, exception and rework rates, cost per transaction, approval latency, and the time required to prepare evidence for an audit or a regulator.

The failure side belongs in the same business case. Rework caused by incorrect records, remediation cost after a disclosure, and the staff time consumed by reviewing low-confidence outputs are not externalities; they are costs the programme creates. Benefits should be reported against a pre-deployment baseline, with the measurement window stated, and success defined as completed work rather than user satisfaction scores.

Questions to put to vendors

• Where is authorisation enforced — in the prompt, or in the API and data layer?
• How is tenant isolation implemented, and can it be demonstrated independently of model behaviour?
• What exactly is written to the audit log, how long is it retained, and can it be exported for a regulator?
• Which actions require human approval, and can we configure the thresholds ourselves?
• How are tool permissions scoped and revoked, and how quickly?
• What happens to in-flight tasks when we suspend the system?
• How do you test for prompt injection through ingested content, and can we see the results?
• Does your evaluation measure output quality, or completed work against system state?

Conclusion

Guardrails do not make AI safe in any absolute sense, and no framework should be presented as though they do. NIST's guidance is voluntary by design, it treats trustworthiness as a set of interacting characteristics rather than a finish line, and it is written to adapt as the technology changes. What well-designed controls do is convert open-ended risk into bounded, observable, recoverable risk — which is precisely the condition under which delegating work to software becomes a rational business decision.

The organisations that capture durable value from AI over the next several years will not be the ones that moved fastest, nor the ones that moved last. They will be the ones that can answer, for every system they operate, what it may read, what it may change, whose authority it carries, who can stop it, and what evidence remains afterwards. Those answers are not obstacles to adoption. They are the terms on which adoption can be defended to a board, a customer and a regulator.