Autonomy needs a boundary, a checkpoint and a way back.
Combine hard policy, eligible tools, confirmation, recovery and evaluation traces so autonomy grows with evidence rather than confidence.
Short answer
Guardrails are product controls around the model, not reminders inside its prompt. Start with a hard policy that defines data access, eligible tools, spending or operational limits and actions that always require a person. Give the model only the choices valid for the current state. Preview consequential changes, confirm them in the host product, recheck policy at execution and design idempotency, cancellation and recovery. Trace tool calls and resulting state, then expand autonomy only where evaluations and operational review support it.
Key points
- 01
Hard limits live in code, credentials and tool eligibility, not in persuasive wording.
- 02
Confirmation is useful only when people can see the proposed effect and still cancel.
- 03
Evaluation should inspect actions and resulting state, not only the final message.
01 / SITUATION
Useful autonomy is a product decision, not a model setting
The same model can answer a procedure, prepare a work order or submit a change. These operations do not carry the same consequences. A single autonomy switch hides the distinctions that matter: data scope, reversibility, financial or operational impact and the ability to detect failure.
A trustworthy design allocates authority by action. The model can have freedom inside a narrow, observable space while the host product keeps the limits, credentials and recovery path. More capability then becomes a controlled extension of the product rather than a broader prompt.
02 / MECHANISM
A layered autonomy contract
Each layer should continue to protect the system when another layer is uncertain or fails.
- 01Classify the outcome
Separate read, draft, propose, execute and irreversible effects. Record who owns each decision and what evidence is required.
- 02Set the hard boundary
Enforce identity, data scope, budgets, invariants and forbidden operations in deterministic code and scoped credentials.
- 03Reduce eligible tools
Expose only the operations valid for the role, workflow and current state. Tool retrieval may narrow this set but never widen it.
- 04Place meaningful checkpoints
Show the target, values, evidence and expected effect before consequential work. Ask for confirmation where the person can still edit or cancel.
- 05Execute with recovery
Revalidate immediately before the effect. Use duplicate protection, timeouts, partial-failure states and a documented path to retry, compensate or escalate.
- 06Evaluate traces and outcomes
Test denied actions, injections, stale state, tool errors and alternate paths. Review the transition trace, tool calls and resulting state alongside communication quality.
03 / In real work
Customer support
The assistant may read permitted account state and prepare a response. A refund or account change remains a typed proposal until policy and an authorised checkpoint allow execution.
Industrial operations
The assistant can find a procedure and prepare an intervention note. It cannot bypass equipment locks, safety rules or the operator responsible for the action.
Narrow experimental loop
Karpathy’s autoresearch repository illustrates one bounded pattern: a single editable file and a fixed evaluation loop. It is a field example for constraining experiments, not proof that the same design is safe for general production work.
Common failure modes
- Telling the model to be careful while giving it broad credentials and tools.
- Letting semantic routing expand a deterministic permission boundary.
- Adding confirmation after the side effect or without showing the exact proposal.
- Granting autonomy without cancellation, duplicate protection or a partial-failure state.
- Evaluating polished transcripts while ignoring tool calls and the state left behind.
Implementation checklist
- Every operation has a consequence and reversibility class.
- Hard policy is enforced outside the model.
- Credentials and eligible tools follow least privilege.
- Important proposals are visible and editable before confirmation.
- Execution rechecks identity, state and invariants.
- Retry, cancellation, compensation and escalation are designed where relevant.
- Traces link decisions, calls, results and final state without leaking secrets.
- Autonomy expands only after representative evaluations and operating review.
What this approach does not claim
- Guardrails reduce risk; they do not make an open-ended agent safe in every environment.
- A human checkpoint can become a rubber stamp if proposals are unclear or too frequent.
- Field examples and vendor guidance must be tested against the organisation’s own tools, threats and responsibilities.
Sources and reference types
- 01Artificial Intelligence Risk Management Framework 1.0NIST · 2023Standard
- 02AgentDojo: A Dynamic Environment to Evaluate Prompt Injection AttacksDebenedetti et al. · 2025Research
- 03autoresearchAndrej Karpathy · 2026Field practice
- 04AI Elements documentationVercel · 2026Official documentation
- 05Trustworthy agents in practiceAnthropic · 2026Field practice
- 06Demystifying evals for AI agentsAnthropic · 2026Field practice
Practical questions
Should every tool call require approval?
No. Read-only and low-consequence operations may run automatically within strict scope. Approval should match consequence, reversibility and evidence.
Is a system prompt a guardrail?
It can guide behaviour, but it is not a hard boundary. Access, tool eligibility, validation and execution policy belong to the host system.
What should an autonomy evaluation verify?
Whether the task outcome is correct, the chosen path is allowed, required checkpoints occurred, restricted actions were refused and failure leaves a recoverable state.