Dossier entry
Demo vs deployment: what a showcase can't tell you
The plain answer
A demo shows that something can happen, under conditions the presenter chose. Deployment shows how often it happens, for ordinary users, on inputs nobody picked in advance. To judge an AI claim, ask how the demo was made, how often it succeeds, and whether people outside the company can try it.
AI announcements increasingly arrive as videos: an assistant books a trip, writes an app, or holds a fluid conversation. Demos are good at showing what is possible. They are poor evidence of what is reliable.
Why demos flatter
A demo is a best case by design. Common, often legitimate, production choices include:
- Selection: choosing the run that worked best from many attempts.
- Editing: cutting out pauses, retries, or errors.
- Scripted inputs: using prompts known to work well.
- Hidden help: human operators, extra instructions, or tools that real users won't have.
- Speed changes: shortening waiting time in the edit.
None of these is necessarily dishonest, especially when disclosed. But each one widens the gap between what you see and what you get.
Questions to ask about a demo
- Is it live or recorded? If recorded, was it edited?
- How many attempts did this take? A success rate is more informative than a single success.
- Were the inputs prepared in advance?
- Was any human involved during the task?
- Can I use the same system myself, today?
- Does the fine print mention speed-ups, scripted prompts, or simulation?
What deployment reveals
Once a system is in many people's hands, different evidence appears:
| Demo shows | Deployment shows |
|---|---|
| That a task can succeed | How often it succeeds |
| Hand-picked inputs | Unpredictable real inputs |
| The fastest case | Typical speed and cost |
| No visible errors | Failure modes and their impact |
| The developer's view | Independent users' and testers' views |
That is why reports from independent testers and ordinary users, and documented failure cases in system cards, matter so much.
A balanced reading
Demos are not worthless. They show direction, and they sometimes reveal genuinely new abilities. The useful habit is to treat a demo as a hypothesis ("this system can do X") and look for evidence that it does X reliably.
Limitations
Judging reliability takes time and access many readers do not have. When you cannot test a system yourself, look for independent evaluations, and weigh the claim with the hype check. Related: How to read an AI benchmark claim.