From pilot to production: putting AI agents to work
Most agent pilots impress in a demo and stall after it. The gap is rarely the model. It is access, guardrails and measurement.
By the NIMTECH engineering team6 min read
Why pilots stall
A pilot usually runs on a copy of the data, with a friendly user and no connection to the systems where the work actually happens. It answers questions well, so everyone is impressed. Then it meets production: real permissions, messy records, edge cases and a risk team that wants to know what happens when it is wrong.
At that point the hard questions are not about the model. They are about which systems the agent may read and write, who approves what, how you know it is getting better or worse, and what it costs per task.
Pick one workflow, not one chatbot
Good first candidates are high-volume, rule-heavy tasks where a person currently copies information between systems: order and billing requests, claim intake, supplier onboarding, ticket triage. Each has a clear owner, a clear definition of done and a number you can move, such as handling time or first-contact resolution.
Write the workflow down step by step before writing any code. The steps an experienced person takes, and the checks they make, become the agent’s tools and guardrails.
Connect it to real systems, safely
An agent that cannot read the CRM or update the ticket is just another tab to check. Give it narrow, purpose-built tools (look up an order, issue a refund up to a limit, draft a reply) rather than broad database access. Each tool enforces its own permissions and writes to an audit log.
Keep humans where the stakes are high. A refund above a threshold, a contract change or a message to a key account goes to a person for one-click approval, with the agent’s reasoning and sources attached.
Measure before you scale
Build an evaluation set from your own historical cases, including the awkward ones. Run it on every change to prompts, tools or models, and track accuracy, cost and latency side by side. When the numbers hold steady in production for a few weeks, widen the scope.