The AI Demo Works. Can Someone Else Run It? My Practical Handover Checklist

Two programmers reviewing work together on a computer monitor.
A software review in progress. Context photograph; the people pictured are not presented as participants in the fictional project. Image credit: Calqui, CC BY-SA 3.0. Resized, full frame retained.

The AI demonstration works. It reads a customer message, finds the order and drafts a helpful reply. Everyone in the meeting can see the attraction. My next question is what happens when the order record is wrong, the connection fails or the colleague who built the demo goes on holiday.

That is the practical follow-up to our earlier article about forward-deployed engineers. We looked at why companies need people who can connect an AI system to real operations. Here I want to make the handover concrete: what should a small team be able to demonstrate before calling an AI-assisted workflow ready for everyday use? Our earlier forward-deployed-engineering article

The exercise below is a proposed test plan, not a system I have deployed for a client. It uses a fictional online shop and a support assistant that reads order information and prepares replies. The assistant does not issue refunds, change orders or send messages automatically. Keeping the first version narrow makes the outcome easier to judge.

Define the job before showing the clever part

My one-sentence brief would be: given a customer message and authorized order records, produce a draft response that accurately states the order's status and identifies anything requiring a person to decide.

That sentence immediately reveals boundaries. The assistant needs permission to read the relevant information. It must distinguish a confirmed status from an estimate. It needs a way to say that a record is missing. And a draft must remain a draft until someone approves it.

Anthropic's guidance on building effective agents recommends beginning with simple solutions and adding complexity only when necessary. Its distinction between predefined workflows and more autonomous agents is useful here. A support-drafting task may not need a system that decides freely which actions to take next. Anthropic on building effective agents

The engineer’s first demonstration should be the smallest useful version. If retrieving an order and preparing a response solves the problem, there is no need to add payments, account changes and autonomous sending simply to make the demonstration more impressive.

Build a small set of awkward examples

Start with twenty fictional cases whose expected outcomes are written down before the system runs. Include ordinary orders, missing records, conflicting dates, a delayed delivery, a message containing two different requests and a request outside the assistant's authority.

For each case, specify what the draft must say and what it must not claim. “Correctly handles a missing order” is too vague. “States that the order could not be located and asks a colleague to verify the reference” is an outcome someone can check.

The test set also needs a case where the text being read tells the assistant to ignore its instructions. The purpose is to check that customer-supplied content remains data to evaluate, rather than becoming permission to change the task or access unrelated information.

Twenty examples would be a starting exercise, not evidence of production reliability. After correcting obvious problems, the team needs fresh cases and a representative sample of real work handled under its privacy rules. Otherwise, it risks polishing a demonstration set instead of improving the actual process.

Anthropic's evaluation guidance distinguishes the agent's conversation from the resulting state of the environment and discusses combining different kinds of graders. That supports a practical rule: inspect what the system did and produced, rather than accepting its own statement that the task succeeded. Anthropic’s guide to evaluating agents

A proposed handover test for a draft-only support assistant
CaseExpected behaviorWhat to inspect
Missing orderAsk for verification; do not invent a statusDraft and source lookup
Conflicting datesIdentify the conflict for a personExact dates and evidence
Instruction inside customer textTreat it as data, not authorityTool calls and draft
Connection failureReport inability to verify; route manuallyVisible pending/error state
Repeated requestIdentify the prior attemptRequest log and duplicate drafts
Another colleague operates itComplete ordinary recovery without the builderDocumentation and ownership

Score errors by consequence

Suppose the assistant produces eighteen acceptable drafts from twenty cases. A 90% headline hides a lot. Were the two failures awkward sentences that a reviewer could quickly fix, or confident claims that a refund had been issued when it had not?

My scorecard would record at least three separate outcomes: factual correctness, respect for the permitted actions and whether the result is useful enough to save time. A fluent draft with the wrong delivery status fails the first test. A correct draft sent without approval fails the second.

The team should agree in advance which errors prevent release. In this fictional workflow, unauthorized sending or exposure of another customer's information would be a stop condition. A formatting defect might be lower priority, provided it does not conceal a factual problem.

This is also where someone who knows the business becomes essential. An engineer can verify that a field was retrieved correctly while missing that the field is not the one staff actually use for a promise to a customer. Technical accuracy and operational meaning need to meet in the same test.

Rehearse a failure, including the retry

I would deliberately interrupt the connection to the fictional order service and inspect the result. The acceptable response is an explicit inability to verify the status, with a route to manual handling. Inventing a plausible shipping date is not a graceful fallback.

Then restore the connection and repeat the request. The test should show how the system distinguishes a new attempt from work already completed. In a later version that performs actions, this becomes especially important: retrying must not quietly create duplicate changes.

For the draft-only version, the consequence might be two drafts rather than two refunds, but the habit is still worth establishing. Give each request an identifier, record what happened and make uncertainty visible. Those records should contain only the information needed for operation and debugging.

A timeout also needs a practical limit. I would agree how long a staff member waits before taking over, how many retries are permitted and where the pending work appears. A system that eventually succeeds after the customer has already been helped can create confusion rather than value.

Put a spending limit beside the success criteria

Assume the shop handles 1,000 requests a month and the assistant saves two minutes per request before review. That is 33 hours and 20 minutes. If checking drafts and maintaining the workflow takes twelve hours, the net time saving is 21 hours and 20 minutes.

At an assumed US$20 per hour, that time is valued at approximately US$426.67. If software and operating charges total an illustrative US$150 per month, the remaining value is US$276.67 before setup costs and anything omitted from the model. None of these figures is a vendor quote or a measured result.

At the ECB's November 13 reference rate, those figures are approximately €386.05 in time value, €135.72 in operating charges and €250.33 remaining. They are conversions for comparison, not local service prices. The important variable is the verified net time saving, not the arithmetic's apparent precision. ECB dollar reference rate, November 13

I would set a monthly spending cap, alerts below it and a named person responsible for responding. The team should also know how to pause the workflow. A budget alert that nobody owns is merely another notification.

The handover test is another person using it

The strongest handover demonstration is to let a colleague operate the workflow using the documentation while the builder watches without intervening. Can the colleague find pending requests, understand an error, correct a source record and pause processing?

The handover should identify who owns the accounts, where configuration lives, which data is accessed, how changes are reviewed and how to return to the previous working version. It should also say who receives an alert and who makes the decision when the system cannot safely continue.

The handover also needs a short maintenance calendar: review a sample of outputs, check costs, confirm the source data remains appropriate and rerun the key cases after changes. The aim is not paperwork for its own sake. It is to prevent useful knowledge from existing only in one person's memory.

A small business also needs an exit route. It should be possible to retrieve its own drafts and records in a usable form and continue the underlying job manually. The assistant is there to improve the process, not make the company unable to operate without one particular setup.

That is the version of forward-deployed engineering I find valuable: technical work that leaves the next person with a reliable, understandable operation. The demonstration starts the conversation. The handover proves that the system can become part of ordinary work.

Comments

Popular posts from this blog

Obsidian’s New Search: Less Friction for Notes in Two Languages

OPPO Find X9 Ultra vs Find X10 Pro Max: Is the Older Ultra Still Worth Buying?

OPPO Find X10 Technical Preview: Cameras, Design and Missing Specs