How to choose your first AI pilot

Pick a deliverable someone can evaluate

A proposal like “use AI in operations” leaves many decisions open. Try describing it as a verifiable deliverable: summarizing a request, finding a procedure or suggesting how to fill in a record. Name who receives that output and what that person needs to check before using it.

We recommend gathering a few opportunities and comparing them on data access, task frequency, clarity of the output and the impact of an error. Favor a scope with an owner who has time to follow the pilot. The project has to fit into the routine of the people who will teach, test and evaluate it.

Describe the starting point

Before implementation, observe real examples of the process. Record the time spent, where work waits, what gets sent back and the systems involved. Also note what already works well. This baseline lets you judge the result without relying on the impression a demo leaves.

Consider a hypothetical example: summarizing tickets for a shift handover. The pilot can deliver a draft with the status, pending items and next owner. The person taking over the shift checks whether they can pick up the work and records the information they had to correct or look up again.

Agree on criteria before showing the first version

An AI evaluation compares inputs and outputs against success criteria. For agents, Anthropic highlights the importance of checking the final state of the task, not just the text produced. The article also notes that responses vary from run to run; repeating trials helps gauge consistency.

Translate this into business terms with written criteria. In the shift handover example, the draft must include the known pending items, identify where each piece of information came from and flag open questions explicitly. The criteria should make sense to the people who do the work, without anyone having to interpret a technical score.

• Quality: which errors make the deliverable unusable?
• Effort: how much review and correction work remains?
• Operations: who monitors failures and updates the content in use?
• Limits: which data, actions and situations fall outside this pilot?
• Decision: what evidence justifies continuing, adjusting or stopping?

Hold back cases to track progress

Set aside some of the examples to compare versions, including incomplete cases and common exceptions. Record what changed from one round to the next. If an improvement makes the summaries shorter, check that the important pending items are still there. The goal is to see the effect of each change in real use.

Agree from the outset on who can halt the experiment and how the routine continues when the tool is unavailable. Make time to listen to the team that receives the outputs. A remark like “I had to check everything again” belongs in the evaluation, even when the text looks correct.

Close the pilot with a documented decision

At the end, gather the examples, corrections, observed costs and maintenance effort. Compare them with the starting point and record the decision: expand use, revise the scope or stop. Financial gains should only be presented when there is a basis for calculating them.

A pilot also delivers value when it exposes a difficult integration, a poorly defined rule or an outdated data source. That lesson should inform the next decision. The next investment rests on firmer ground when the expected outcome and the known limits are written down.