Why Agent Pilots Stall Before Production

The demo was not the hard part

Building an agent that reads a ticket, looks up a policy, checks a record and drafts a reply is now a short piece of work. A competent engineer with access to a model API and a few internal endpoints can have something convincing inside two weeks. That is why so many organisations have three or four of them.

The demo is convincing because of what it is permitted to skip. It runs under a developer's credentials, on a handful of cases chosen because they are clean, with the engineer watching the output. Nothing is logged in a form an auditor would accept, nothing retries, nothing rolls back, and nobody is on call. Every one of those omissions is a separate piece of engineering and organisational work, and together they are usually larger than the agent itself.

So the pilot does not fail. It stalls. The business owner says it works, the security team will not sign it off, and nobody can produce a plan that converts the first into the second. Months pass. The model version it was built against is deprecated. The engineer moves teams.

This paper covers the specific things that cause that stall, in roughly the order they surface during a real rollout. None of them are research problems. All of them are decisions somebody has to make and write down.

The permission problem

This is the one that stops most pilots, and it is almost never framed correctly. The usual framing is "the agent needs access to the CRM". The accurate framing is: an agent that can read everything a user can read inherits an access model that was designed for a human with judgement, working at human speed, under social and managerial constraint. Automation removes all three of those.

Consider what your existing access model quietly relies on. A support agent can technically open any customer record in the system, and does not, because they have no reason to and because someone would notice. A finance analyst can export the full ledger, and does not, because the export is tedious and the folder is shared. These are not controls. They are frictions that have been doing the work of controls for years. An agent with the same entitlements has no boredom, no reputation and no manager, and it will exercise the full extent of its permissions whenever a prompt or a retrieved document steers it there.

Three distinct problems hide inside "give the agent access":

Breadth. Service accounts tend to accumulate entitlements because it is easier to grant than to scope. If the agent authenticates as itself, its identity typically ends up broader than any single user's, because it must serve all users. The practical consequence is that a prompt-injection payload in one customer's uploaded document can reach another customer's record, and the system will consider that authorised.

Delegation. The alternative — the agent acts as the requesting user, carrying their token — is usually the right default, and it is harder than it sounds. It requires on-behalf-of token exchange through every hop, including the retrieval layer. If your vector index was built by a batch job with broad read rights and does not carry per-document ACLs, permission-aware retrieval is not something you can switch on later. It is a re-index.

Write scope. Read access leaks. Write access does damage, and the two need completely different treatment. Every tool the agent can call should be declared individually, with its own scope: not "update CRM" but "update the next-action field on an opportunity owned by the requesting user, where the opportunity is in the ticket referenced in this conversation". That specificity is what makes a security review short.

A useful test before you go any further: ask for a written list of every tool the agent can invoke, every data source it can read, and the identity under which each call is made. If producing that list takes more than a day, the permission model does not exist yet — it is an emergent property of whatever credentials were convenient during the build.

The audit trail you will need six months from now

At some point, somebody will ask you to explain one specific decision. It will be a customer disputing an outcome, an internal auditor sampling a quarter's transactions, or a regulator following up on a complaint. It will be months after the fact. The model version will have changed. The prompt will have been edited. The retrieved document will have been superseded.

Most pilots log the request and the response. That is not enough to answer the question. You need to be able to reconstruct the entire input that produced the output, because in a retrieval-based system the inputs are assembled at runtime and no two runs are identical.

At minimum, capture per interaction:

• The requesting identity and the identity used for each downstream call, so delegation is provable.
• The user input verbatim, and any system or developer prompt by content hash plus a version reference — so you can prove which prompt was live without storing the full text on every record.
• The model and version string returned by the provider, not the one you asked for. Aliases move.
• Every retrieved chunk: source document identifier, version, retrieval score, and the ACL decision that permitted it. This is the field most often missing and most often needed.
• Every tool call with its arguments, the result, latency, and whether it was a read or a write.
• Confidence or guard-rail signals, and any refusal or escalation, with the rule that triggered it.
• The human decision, if there was a checkpoint: who, when, what they saw, and whether they modified the proposal or accepted it unchanged.

Two engineering notes save pain later. Propagate a correlation identifier into the logs of every system the agent touches, so a record change in the ERP traces back to the conversation that caused it — investigations start from the damaged record, not from the agent log. And decide retention before go-live. Agent logs are the richest collection of personal data in the stack, and a verbatim log kept indefinitely is a DPDP problem you created while solving an audit problem. Separate the forensic trail from the content: keep identifiers, hashes, decisions and ACL outcomes long, keep raw content for a short, justified window.

interaction_id: 8f2c-...-91ab        correlation_id: trace-4471
requested_by:   user:4821 (role: claims_assessor_l2)
acting_as:      user:4821  (obo token, aud: claims-api)
model:          vendor-model-x, version 2025-08-14 (as returned)
prompt_ref:     claims-triage/v7  sha256:3d9e...
retrieved:
  - doc:POL-2219 v4  score 0.81  acl: allow (role match)
  - doc:CIR-0087 v2  score 0.74  acl: allow (role match)
  - doc:POL-3310 v1  score 0.69  acl: DENY  (business unit mismatch) [excluded]
tools:
  - policy_lookup(policy_id=POL-2219)            read   140ms  ok
  - claim_set_recommendation(claim=C-55120, ...)  WRITE  210ms  ok
guardrails:     amount_threshold_exceeded -> human_checkpoint
human:          user:1190 approved 2025-09-02T11:40Z, modified 1 field
content_retention: 90d    trail_retention: 7y

Human checkpoints that do not become rubber stamps

Everyone agrees on human-in-the-loop. Very few designs survive the second month, because the way most checkpoints are built guarantees they degrade into clicking approve.

A checkpoint degrades when three conditions hold: it fires on almost every item, the agent is almost always right, and reviewing properly takes longer than the reviewer has. Under those conditions a rational person approves by default, and the control now exists only on the architecture diagram. Worse, you have created a liability: the organisation believes a human reviewed the decision, and formally one did.

What makes checkpoints hold:

• Put them where the action is hard to reverse, not where the model is uncertain. Uncertainty routing is useful, but reversibility is the better primary criterion. A payment, an external send, a record change that cascades, a customer-facing commitment — these get checkpoints regardless of confidence. A draft that a person will read anyway does not need one.
• Keep the volume low enough to review properly. If more than roughly a fifth of items reach a human, the checkpoint is doing triage, not oversight. Narrow the criteria, or accept that the workflow is not ready for automation.
• Show the evidence, not just the proposal. The reviewer needs the retrieved passages, the rule that applied, and the specific fields that will change. An interface that shows only the conclusion is an interface designed to produce approvals.
• Make rejection cheap and informative. A reviewer who rejects should be able to say why in one click from a short list. Those reasons are your best evaluation data and your earliest drift signal.
• Measure the checkpoint itself. Track approval rate, median review time and modification rate per reviewer. An approval rate near 100% with a review time under a few seconds is a rubber stamp, and you should treat that as an incident rather than as success.

One uncomfortable consequence: a well-designed checkpoint sometimes shows that the automation saves less time than projected, because the review is the expensive part. It is better to learn that in week six than to discover it after the business case was signed.

Evaluation that survives the real data distribution

The twenty examples used during the build are not an evaluation set. They are the set the system was fitted to, selected — usually unconsciously — because they were legible. Real production traffic contains the cases nobody thought to put in a demo: the scanned document rotated ninety degrees, the ticket written in three languages, the policy that was superseded last quarter but still sits in the index, the customer whose record was migrated and has two identifiers.

Building an evaluation set that means something is tedious, unglamorous and the single highest-value thing you can do between pilot and production. Sample from production reality rather than curating: a stratified sample of real historical cases, common ones in proportion, plus deliberate over-sampling of the categories your operations team complains about. Ask the people who do the work today which cases they dread; those are your hard set. Include cases where the correct answer is "escalate" or "I cannot answer this", because refusal behaviour is a capability and most pilots never measure it.

Then decide what you are measuring, per workflow, with a number attached before you run anything:

• Task correctness against a human-labelled answer, scored by someone who does the job. Agree the labelling rules in writing; two assessors will disagree on edge cases and you need to know by how much.
• Groundedness — whether each claim in the output is supported by something retrieved. An answer that is correct by luck is a failure you have not detected yet.
• Harmful-error rate separately from overall error rate. A wrong recommendation that a reviewer catches and a wrong write that reaches the core system are not the same event and should never share a metric.
• Refusal and escalation behaviour on the cases where escalating is correct, and on the cases where it is not. Over-escalation is the failure mode that quietly destroys the business case.
• Robustness against adversarial inputs: prompt injection in retrieved documents and user uploads, contradictory instructions, attempts to make the agent act outside its declared tools.

Keep a held-out slice you do not look at while tuning, and re-run the whole set on every change to model version, prompt, retrieval configuration or index content. Automate the run; if it takes a person a day, it will stop happening by the third month. Then keep sampling production after go-live — a fixed percentage of live interactions reviewed weekly — because your input distribution will move and the evaluation set you froze at launch will slowly stop describing your traffic.

Wrong but valid-looking: failure handling and rollback

Classical software fails loudly. It throws, times out, returns a 500. Agents have a failure mode conventional operations tooling was not built for: a confident, well-formed, plausible action that is wrong. The tool call succeeded. The schema validated. The record was updated. The field now contains a reasonable-looking value that is not correct. Nothing in your monitoring will catch that, because by every signal the system is healthy. Design for it explicitly.

• Make writes idempotent and attributable. Every write carries the interaction identifier. You must be able to ask a system of record "show me everything the agent changed between these two timestamps" and get a complete answer in minutes, not from a log archaeology project.
• Prefer proposals to direct writes where the domain allows it. Writing to a staging table, a draft state or a pending-approval queue converts a rollback problem into a discard problem.
• Write the compensating action before you ship the action. For each write the agent can perform, document how it is reversed, whether reversal is automatic or manual, and what downstream effects it cannot undo — notifications already sent, a payment file already released, an SLA clock already started.
• Bound the blast radius with rate and volume limits. An agent in a bad loop can touch thousands of records before anyone notices. A cap on writes per hour, per entity and per run is the cheapest control you will ever add.
• Build a kill switch that someone other than the build team can operate. A feature flag that disables the agent's write tools while leaving it readable, documented in the runbook, tested in a drill. If disabling the agent requires a deployment, you do not have one.
• Define degraded mode. When the model endpoint is slow, the index is stale or the guard-rail service is down, what happens? Falling back to the old manual queue is a legitimate and often correct answer, but only if the queue still exists and someone is still trained on it.

Run one deliberate failure drill before go-live. Inject a wrong action into a test environment, then have the operations team find it, scope it, reverse it and report it using only the runbook. The gaps that surface in that two-hour exercise are the ones that would otherwise surface during a real incident.

Ownership after go-live: who gets paged

This is the least technical section and the one most pilots have no answer to. An agent in production runs unattended, touches systems of record, and can be wrong in novel ways. It needs the same operational treatment as any other production service, plus two things ordinary services do not have: a quality owner and a model-change process. Four roles need names against them before go-live. Not teams — names.

Role	Owns	Called when
Business owner	Whether the workflow outcome is acceptable; the decision to widen or narrow autonomy	Quality degrades, exception volume moves, the business case needs revisiting
Technical on-call	Availability, latency, errors, cost, the kill switch	Standard production alerting; model endpoint failures; cost anomalies
Quality owner	Evaluation runs, sampled production review, drift detection, prompt and retrieval changes	Evaluation regression, groundedness drop, reviewer rejection-rate change
Security contact	Permission changes, incident classification, prompt-injection findings, re-test schedule	Suspected injection, unexpected data access, any entitlement change request

Alongside the names you need three artefacts. A runbook that covers the kill switch, degraded mode, the rollback procedure and the escalation path, written so someone who did not build the system can follow it at 2 a.m. A change process stating who may edit a prompt or swap a model version, and that no such change ships without an evaluation re-run — prompts are production code and the most common cause of silent regression is an untracked prompt edit. And a monthly report, produced from the logs rather than assembled by hand, covering volume, escalation rate, error rate, cost and incidents.

If no team will accept the pager, that is a decision about the agent, and it is worth hearing it clearly. An unowned agent in production is a liability with a plausible interface.

The security review, and how to pass it

Most pilots that die in security review die because they arrive with the wrong artefacts. The team presents a demo and a description of business value. The reviewer needs to answer a specific set of questions and cannot, so the answer is no — and the no feels arbitrary to the build team, which then concludes that security is blocking AI. It is not arbitrary. The questions are predictable, and they come in roughly this order.

What can this thing do? The declared tool list, with scopes. Not capabilities in prose — the actual enumerated set of calls, their parameters, their blast radius, and which are writes.

Whose authority does it act under? The identity model. Delegation or service account, token audience and lifetime, and how the retrieval layer enforces the same entitlements as the source systems. A vector index without per-document ACLs will be found, and it should be.

What happens when the input is hostile? Prompt injection is not hypothetical for any agent that reads documents, emails or tickets from outside the trust boundary. The reviewer wants evidence of testing: injection attempts embedded in retrieved content, attempts to make the agent exceed its tool scope, attempts to exfiltrate context through an allowed output channel. Findings and fixes, with severities. Showing that you found and fixed three issues is far stronger than claiming there were none.

Where does the data go? A data-flow map covering every hop, including the model endpoint. Region, retention, whether prompts and outputs are retained by the provider, whether anything is used for training, and the contractual basis for each answer. For DPDP purposes the personal data categories in prompts and retrieved context need naming, along with the lawful basis and the retention decision discussed above.

Can you prove what it did? The audit trail, with a worked example. A reviewer will often pick one historical interaction and ask you to reconstruct it. If you can do that in front of them, the conversation changes.

How do you stop it? The kill switch, who can operate it, how fast, and when it was last tested.

Who is responsible? The four names above.

The practical advice is simple and widely ignored: involve the security reviewer at design time, not at sign-off. A thirty-minute conversation in week two about the identity model and the retrieval ACLs will change the architecture in ways that are cheap then and expensive later. A reviewer who shaped the design is a reviewer who already understands it.

The production-readiness checklist — and the pilots that should fail it

Use this as a gate, not a wish list. Each line has a pass criterion that is either met or not met; "in progress" is a fail. We score these with clients before go-live, and a pilot that fails more than two lines in the first three sections is not close to production regardless of how well the demo performs.

Area	Check	Pass criterion
Permissions	Tool inventory	A written list of every tool, its parameters, its scope and whether it writes. Produced in under one day, reviewed by the system owner of each target system.
Permissions	Identity model	Agent acts as the requesting user, or the deviation is documented and signed off by the security contact. Token audience and lifetime specified.
Permissions	Retrieval entitlements	Retrieval filters by the requesting user's entitlements at query time, verified by a test where two users with different rights ask the same question and get different results.
Permissions	Least privilege on writes	No tool can write outside the entity referenced in the current task. Verified by an attempted out-of-scope write that fails.
Auditability	Reconstruction test	Pick any interaction from the last 30 days and reconstruct inputs, retrieved context, ACL decisions, tool calls and human actions in under 15 minutes.
Auditability	Correlation	A change in a target system can be traced to the originating interaction using that system's own logs.
Auditability	Retention and redaction	Separate retention periods set for the forensic trail and raw content, with a documented lawful basis. Deletion actually executes — tested once.
Evaluation	Set size and provenance	At least 200 cases sampled from real historical traffic, stratified, including a hard set chosen by the operations team and cases where refusal is correct. Held-out slice untouched during tuning.
Evaluation	Thresholds agreed in advance	Task correctness, groundedness, harmful-error rate and over-escalation rate each have a go/no-go number agreed with the business owner before the run.
Evaluation	Adversarial testing	Prompt-injection and tool-abuse testing completed by someone who did not build the agent. Findings closed or accepted with sign-off.
Evaluation	Automated re-run	Full set runs on a single command and is wired into the change process for prompts, models, retrieval config and index updates.
Human oversight	Checkpoint placement	Every irreversible action has a checkpoint. Checkpoint volume under 20% of items, or the gap is explicitly accepted.
Human oversight	Reviewer interface	Reviewers see retrieved evidence and the exact fields changing, and can reject with a structured reason in one click.
Human oversight	Rubber-stamp monitoring	Approval rate, review time and modification rate reported per reviewer, with a defined threshold that triggers investigation.
Failure handling	Kill switch	Non-deployment switch that disables write tools, operable by on-call, tested within the last 30 days.
Failure handling	Rollback plan	Documented compensating action for every write tool, including the irreversible downstream effects, with one drill completed.
Failure handling	Rate and volume caps	Writes capped per hour, per entity and per run, with an alert on approach.
Failure handling	Degraded mode	Defined behaviour when the model, index or guard-rail service is unavailable, including whether the manual fallback still exists and is staffed.
Ownership	Four names	Business owner, technical on-call, quality owner and security contact named, and each has acknowledged in writing.
Ownership	Runbook	Written and validated by someone outside the build team performing a dry run.
Ownership	Change control	Prompt and model changes go through version control and trigger an evaluation re-run. No exceptions path.
Ownership	Cost attribution	Token and infrastructure spend attributable to this workflow, with a monthly budget and an anomaly alert.
Security	Design-time review	Security reviewer participated in the design, not only the sign-off.
Security	Data-flow map	Every hop documented including the model endpoint, with region, retention, training-use and the contractual basis for each.
Security	Re-test schedule	A date set for the next adversarial re-test, and a trigger list of changes that force an early one.

And the pilots that should fail it

We will say this plainly because very few vendors will. A meaningful share of agent pilots should be stopped rather than hardened, and recognising that early is a good result rather than a failed project. The patterns that should stop:

• The value was in the demo, not the workflow. When you measure the actual time spent on the task, including the review the agent now requires, the saving is marginal. This is common in workflows where the expensive part was always the judgement, not the typing.
• The checkpoint eats the benefit. If the action is irreversible enough to need review on every item, and reviewing takes nearly as long as doing, the agent has moved the work rather than removed it. Sometimes the honest answer is a better interface for the human, not an agent.
• The underlying data is not ready. If the policy corpus contains three contradictory superseded versions and nobody owns it, the agent will confidently cite the wrong one. Fix the corpus first; that is a records project, and it is worth doing on its own merits.
• The permission model cannot be made specific. Some legacy systems offer only all-or-nothing access. If the agent cannot be scoped below "everything", and the data class is sensitive, the correct answer may be to wait for an integration layer rather than to accept the exposure.
• No one will own it. If no team will take the pager after a frank description of what that means, the organisation has told you its real appetite. Believe it.

The constructive version of stopping is usually not abandonment. It is narrowing: the same agent, read-only, drafting into a queue a person already works through, is often a genuinely valuable system that passes every check in the table above in a fraction of the time. Narrow autonomy, prove it with evidence for a quarter, then widen it deliberately. That sequence is slower on a slide and faster in reality.

We do not publish invented case studies or success percentages, so we will not tell you what proportion of pilots we have advised to stop. What we will do on a scoping call is walk your agent against the checklist above and tell you which lines fail, what each one costs to fix, and whether the workflow is worth the fix. If the answer is that it is not, you will hear that on the call.