From Pilot to Production: The 90-Day Agent Deployment Roadmap

The conference room had been booked for the third time. The deck was largely the same as the one six months earlier. The pilot had run. The results were positive. And yet the question at the end of the presentation was identical to the one that had delayed the previous two approvals: "How do we know when it's ready for production?"

This scene plays out in enterprises across every regulated industry. Not because the technology is insufficient. Not because the risk appetite is genuinely too low. But because nobody wrote down what "ready" means before the pilot started. When readiness is undefined, it defaults to a subjective judgment that shifts with every presentation, every leadership change, and every new concern raised in the room. The pilot never advances because advancement has no trigger.

This article provides the operational framework for escaping that loop. The three-phase model — Sandbox, Constrained Production, Expanded Autonomy — with specific, measurable gate criteria at each transition, aligned to the NIST AI Risk Management Framework.

What is Phase 1 (Sandbox) and what are its exit criteria?

Phase 1 is a controlled evaluation environment using synthetic or fully anonymized data, with zero exposure to production systems, real user data, or consequential decision pathways. The purpose is to establish baseline performance metrics under controlled conditions. Exit criteria are quantitative: the agent must achieve a defined error rate threshold, a defined latency profile, and zero policy violations across a minimum 500-task evaluation set. Duration is typically 2–3 months. Exit is authorized by a documented sign-off from the AI risk owner, not by team consensus.

The Sandbox phase maps to the NIST AI RMF's Map and Measure functions. The Map function means documenting the specific risks of the AI use case before evaluation begins. The Measure function means defining the specific metrics and thresholds that will be used to evaluate those risks and establishing the evaluation methodology. Organizations that skip the Map function produce Sandbox results that cannot be evaluated against any standard, which is itself a cause of pilot purgatory.

A critical Sandbox design principle is adversarial testing. The evaluation must include not only typical-case inputs but also adversarial inputs: prompt injection attempts, boundary-condition inputs designed to cause scope violations, and inputs from underrepresented categories that may surface bias. An agent that performs excellently on average-case inputs but fails on adversarial inputs is not Sandbox-complete.

What is Phase 2 (Constrained Production) and how is it different from a pilot?

Constrained Production is a live production deployment with hard technical guardrails that enforce defined boundaries — the agent cannot operate outside its sanctioned scope, cannot access unauthorized data, and cannot execute high-consequence actions without a human approval step. It is distinct from a pilot in that it operates on real data, real users, and real workflows, but with guardrails that prevent catastrophic failure modes. Gate to Phase 3 requires a sustained 30-day clean run (zero policy violations, error rate within threshold) and a formal Tier assessment.

The Phase 2 guardrail design is the most technically demanding element of the roadmap. Guardrails must be enforced at the infrastructure layer, not at the application layer. An agent that is "told" not to access certain data through prompt instructions is not guardrailed — it is instructed. Actual guardrails intercept the agent's tool calls and API requests and enforce access control based on the agent's identity and declared task context. A system message is not a SOX audit trail.

The 2–4 month duration recommendation for Phase 2 is based on the need to observe the agent across sufficient volume and variety to establish statistical confidence in its performance profile. An agent that has processed 50,000 tasks across a full business cycle — including month-end processing, regulatory reporting periods, and seasonal volume spikes — has demonstrated performance robustness that a shorter evaluation cannot provide.

What is "Expanded Autonomy" and which agents should never reach it?

Expanded Autonomy (Phase 3) means the agent operates without a mandatory human review step for each individual decision, relying instead on ongoing monitoring and periodic human review of aggregate patterns and flagged exceptions. Tier 1 high-stakes agents — those where an individual wrong decision causes direct, material harm to a person or organization — should not reach Phase 3. This includes agents making credit decisions, clinical documentation, legal filings, regulatory disclosures, and financial transactions above defined thresholds.

This is one of the most important conceptual resets in the AI governance conversation. The narrative of a "journey toward full autonomy" implies that human oversight is temporary scaffolding that will eventually be removed. For the highest-stakes agent classes, this is wrong. The purpose of human oversight is not to compensate for AI immaturity — it is to maintain accountability for consequential decisions in domains where errors have direct human costs. That accountability requirement does not diminish with model performance improvements.

How does the NIST AI RMF lifecycle map to the three phases?

The NIST AI RMF's four functions map to the phases as follows: Govern establishes the organizational accountability structure and risk tolerance before any phase begins. Map occurs primarily in Phase 1 (Sandbox), identifying the specific risk categories for this AI use case. Measure occurs across Phase 1 and Phase 2, quantifying risks against defined thresholds. Manage occurs in Phase 2 (Constrained Production), where identified risks are addressed through guardrails, oversight workflows, and incident response procedures.

The most important practical implication is that the Govern function must be established before the pilot begins — not during or after. Govern means: who is accountable for this AI system's outcomes? Who has the authority to approve phase advancement? Who must be notified if performance degrades below threshold? These questions must be answered in writing, with named individuals and defined escalation paths, before the Sandbox evaluation begins.