Strategy
From pilot to production: a field guide for shipping AI that lasts
The demo works. The room is impressed. Then the pilot ends, the team disperses, and the workflow returns to the way it was.
That gap is not a failure of the model. It is a failure to treat production as a different problem. A pilot proves that a model can produce a plausible answer. Production proves that an organization can rely on that answer on a busy Tuesday, at volume, with an audit trail, and with someone accountable when it is wrong.
The failure modes are predictable. Teams that plan for them ship systems that last. Teams that ignore them spend a year rebuilding the same prototype.
Why pilots stall
The reasons are usually organizational as much as technical, and they tend to appear in a familiar order.
No clear owner
A pilot can survive on borrowed enthusiasm. A production system cannot. If no business leader owns the workflow, no one can decide what “good enough” means, approve a change to the process, or defend the budget when the novelty fades. The pilot team ships a prototype and waits for an owner who was never named.
No evaluation
Pilots are often judged by demonstrations. A room watches a handful of prompts, the answers look right, and the project is declared a success. Without a fixed set of test cases and agreed acceptance criteria, no one can say whether the next model, prompt or retrieval change makes the system better or worse. Teams optimize by anecdote, and quality drifts.
Data access arrives late
The pilot reads a spreadsheet exported for the occasion. Production needs live data, the same permissions as the people using it, a refresh schedule and a retention policy. Each requirement touches systems the pilot never had to integrate with. Discovering this after the demo turns one project into two.
Security review happens at the end
When the architecture is fixed before review, the review becomes a negotiation. Data flows, vendor terms, logging and access control are easier to shape at the start than to retrofit. Bring security and privacy into the first design session, not the last gate.
No operating model
A demo has no on-call rotation. Production does. Someone must watch quality, cost, latency and failures; update prompts and models; respond to incidents; and decide when to retire a feature. If that work has no owner and no allocated time, the system decays from the day it launches.
| Dimension | Pilot | Production |
|---|---|---|
| Question | Can the model do this? | Can the organization rely on this? |
| Success | A convincing demonstration | Measured quality on agreed cases |
| Data | A one-time export | Live access with permissions and retention |
| Security | Reviewed at the end | Designed in from the start |
| Change | Prompts edited by hand | Versioned, tested and reviewed |
| Operations | No owner after handoff | Named owner, monitoring and on-call |
| Cost | Estimated or ignored | Tracked per workflow and reviewed |
Choose the first use case for production, not for the demo
The best demo is often the worst first production system. A general assistant that answers anything is impressive on stage and difficult to evaluate, secure and improve. A narrower workflow is less exciting and far more likely to ship.
Use these tests to choose the first candidate:
First-use-case checklist
- Frequency. The workflow happens often enough that improvements matter.
- Bounded scope. Inputs and outputs are clear enough to test.
- Tolerable failure. Mistakes can be caught by a person or a rule before they cause harm.
- Accessible data. The documents and systems already exist in a usable form.
- A measurable baseline. You can describe how the work is done today and what better means.
- A named owner. A business leader will use the result and fund the next phase.
If a candidate fails two or more of these tests, it is a research project, not a first release.
Define success and evaluation before you build
Evaluation is the discipline that turns an opinion into evidence. It starts with a set of real cases, drawn from the workflow with the consent of the people who own them. For each case, the team records what a good answer looks like, which sources are acceptable, and which failures are unacceptable.
A practical evaluation set
Start with a few dozen cases and grow the set as the system meets real traffic. A useful set covers routine work, edge cases and known traps. It includes the questions users actually ask, not only the ones that are easy to answer.
For each case, write down:
- the expected answer, or the criteria for a good one;
- the sources that count as authoritative;
- the behavior that is unacceptable, such as inventing a citation or ignoring a permission;
- the person who can arbitrate when reviewers disagree.
Review the set before the first release and again after every serious failure. Each failure becomes a new test case, and each new capability gets cases that would catch its characteristic mistakes. This is unglamorous work, and it is the difference between a system that improves and one that merely changes.
Measure more than accuracy
Quality is not one number. Track accuracy, grounding in sources, latency, cost per completed task and the rate at which people accept the output. A system that is slightly less accurate but fast enough to use may be the better product. A system that is accurate but slow enough to be abandoned is not a product at all.
Design for production from day one
Production architecture is mostly about what happens around the model. The model is one dependency; the system around it is what makes the dependency safe to use.
- Retrieval that respects permissions, so an answer never cites a document the user cannot open.
- A model gateway that centralizes keys, quotas, retries, fallbacks and logging.
- Versioned prompts and configurations, reviewed like code.
- Separate environments for development, testing and production.
- Traces that record the question, the sources retrieved, the model and the response.
- Cost controls: budgets, rate limits and alerts before the invoice arrives.
- A fallback path when the model is unavailable or confidence is low.
- Data retention rules that match the promise made to users.
None of this is exotic. It is the same discipline we apply to any system that touches customer data. The mistake is to treat AI as an experiment that will somehow acquire these properties later.
Treat the model as a replaceable part
Models improve, prices change and vendors retire versions. A production system should let you swap the model behind a stable interface without rewriting the application. Keep prompts, retrieval and evaluation outside vendor-specific code, and test each candidate model against the same cases. The team that can change models cheaply is the team that keeps getting better.
Plan operations and ownership
Launch is a beginning. The operating model should name who does what before the first user arrives.
- A product owner accountable for outcomes and priorities.
- An engineering owner accountable for reliability and change.
- A review process for prompt, model and retrieval changes.
- A monitoring routine with thresholds for quality, latency, cost and errors.
- An incident path for harmful, wrong or unavailable outputs.
- A quarterly decision to continue, expand or retire.
Write this down. A one-page operating agreement is worth more than an architecture diagram no one maintains.
Iterate with real users
Release to a small group, in shadow mode where possible, and compare the system’s output with what people actually do. Watch where they ignore it, correct it or work around it. Those moments are the real specification.
Resist the temptation to add features before the core workflow is trusted. A system that does one job reliably earns the right to do more. A system that does many jobs adequately earns nothing.
Set a cadence: weekly review of failures and feedback, monthly review of cost and usage, quarterly review of the use-case portfolio. Retire what does not earn its place.
Summary
Key takeaways
- A pilot proves a model can answer; production proves an organization can rely on the answer.
- Choose a narrow, frequent workflow with a named owner and a measurable baseline.
- Build the evaluation set before the product, and run it before every release.
- Treat permissions, logging, cost and fallbacks as design requirements, not launch chores.
- Name the operating model — owner, review cadence, incident path — before the first user arrives.
If your team has a pilot that will not move, the problem is usually not the model. We help leaders choose the right first use case, design the evaluation and operating model, and build the production system around it. Start with AI strategy or generative AI applications; when the platform itself needs work, our cloud and MLOps practice takes it from there. Tell us where you are today and we will give you a candid view of what is worth building.
Continue reading
More field notes.
Start a conversation
Let’s turn AI ambition into systems you can rely on.
Tell us where you are today. We reply within one business day with a candid view of what is worth building — and what is not.
Or call +1 (714) 900-0052