An AI tool can make a support agent's reply appear almost instantly while the customer still waits for a refund. The distinction is simple: reply speed measures one task; correct resolution measures the whole service. The fictional store in HBW-006 illustrates a common decision error without claiming that every AI pilot fails.
The workflow test
Map one request from arrival to completion. Record each handoff, the rule used to decide, the data required, and the time spent waiting. Then measure three outcomes before introducing a model:
| Measure | What it catches |
|---|---|
| Time to correct resolution | Whether the customer problem is actually closed |
| Repeat-contact rate | Rework caused by unclear or wrong answers |
| Agent effort including review | Time saved in drafting versus time added in checking |
If a policy is inconsistent, clarify it before asking a model to apply it. A small manual review of recent cases can expose disagreements and missing data. Thirty cases is an illustrative pilot size in the episode, not a universal rule. A clear form or simple routing rule may solve some issues without AI.
The evidence is more nuanced than “AI makes support faster.” In the 2023 working-paper version, Brynjolfsson, Li and Raymond studied access to an AI assistant in one customer-support organization and reported a 14% average increase in issues resolved per hour, with larger gains for less experienced workers. That is a meaningful finding in its studied setting, not a guarantee for an unrelated company. Read the research.
NIST's AI Risk Management Framework and Generative AI Profile provide voluntary guidance for defining context, measuring performance and managing risks. A practical pilot should include difficult cases, track rejected suggestions, keep escalation available and name the person who can stop the rollout.
Use this question before the next tool purchase: “This workflow succeeds when ____.” Fill the blank with the customer result, then find the step limiting that result.