What happens when the AI gets it wrong?

Someone said to me, mid-demo, looking at a screen: did this hallucinate or what?

It had. Not badly, but it had. And the fact that we caught it in ten seconds is the entire point of this post.

It will be wrong sometimes

Anyone telling you otherwise is selling. Language models produce confident, well-formed output whether or not they have the right answer, and confidence is not accuracy. That is a property of the technology, not a bug someone will patch.

A client put the sane version of this to me: you also have the extra overhead of double-checking to make sure they are not hallucinating. Correct. If a tool saves you an hour and costs you an hour of checking, you have bought nothing.

So the design question is not "will it be wrong"

It is: what does being wrong cost, and how fast do we find out?

• Drafting an email a human sends — wrong is cheap, and obvious immediately.
• Summarising a document alongside a link to the source — wrong is cheap, because checking takes seconds.
• Posting a figure into your accounts unsupervised — wrong is expensive and invisible. Do not build this.

What I build in

Citations, so every answer points at where it came from. A confidence threshold, so uncertainty escalates to a person rather than guessing. And a log of every run, because someone eventually asks what the system did in March and "I do not know" is not an answer.

One client wanted exactly that: the ability to interrogate everything afterwards for auditing, so if it had got one or two wrong you could find them. That is not paranoia. That is how you deploy this responsibly.

The rule I hold to

If a mistake would be expensive and nobody would notice for a week, a human stays in the loop. No exceptions, and no amount of client enthusiasm changes it.