1. Does it survive unscripted input?
Say:
“Let me type something weird.”
Change:
- Grammar
- Spelling
- Format
- Language
- Order
- Use nonsense
Real system:
Handles ambiguity, fails gracefully.
Fake system:
Breaks, resets, or suddenly “refreshes”.
2. Where exactly is the intelligence?
Ask:
“Which part is AI and which part is normal software?”
Good founders point to:
- Retrieval
- Ranking
- Scoring
- Routing
- Validation
- Decision logic
Bad founders point at:
- The screen.
3. Is it dependent on perfect input?
If the demo:
- Needs copy-pasted text
- Pre-filled data
- Ideal conditions
- Founder typing
…you’re watching demo magic, not a product.
4. What happens when it’s wrong?
Ask:
“Show me a failure case.”
Real founders:
- Willingly show errors
- Explain why
- Show safeguards
Fake founders:
- Avoid showing failure
- Claim high accuracy
- Change topic
If there’s no error handling,
there’s no intelligence.
5. Can it be broken by refreshing the page?
Ask:
“What persists across sessions?”
If memory or state vanishes on refresh:
- It’s either:
- a stateless wrapper
- or caching tricks.
Real systems know what happened earlier.
6. How fast is it?
Watch:
- Latency
- Thinking pauses
- Weird delays
If it exactly mirrors ChatGPT speeds:
- That’s a clue.
Real architecture usually shows:
- Different response profiles
- Model routing
- Retrieval delays
7. Ask about evaluation
“How do you measure success?”
Green flags:
- Ground-truth comparisons
- Error analysis
- Confidence scoring
- Monitoring dashboards
Red flags:
- “Users love it”
- “Adoption is great”
- No metrics
If nobody measures it,
nobody controls it.
8. Who operates it?
Ask:
“What breaks at 2 AM?”
If answer =
“We’ll handle it.”
Run.
Real companies talk about:
- Alerting
- Failovers
- On-call systems
- Automated recovery
9. Can it explain itself?
Say:
“Why did it give this answer?”
If answer is vague or handwavy:
- No reasoning layer exists.
Explainability isn’t optional.
It’s architecture.
10. Kill the internet. Seriously.
Ask:
“What happens if your API goes down?”
If:
- The demo dies.
- Chats vanish.
- Features evaporate.
It’s an API hostage system.
Bonus Trick (use this once quietly):
Ask them to:
- Run the demo from a laptop you choose.
- Or hot-spot your own phone.
If they flinch,
you’ve caught theatre.
The Demo Smell Index
Count how often you hear:
- “Roadmap”
- “Soon”
- “Next quarter”
- “Coming feature”
- “Enterprise”
- “Scale”
vs:
- “Bug”
- “Latency”
- “Tradeoff”
- “Constraint”
- “Limit”
- “Fail”
If right column = zero, it’s fiction.
Final Rule:
Demos should make you uncomfortable.
If you never see fragility, it’s fake.
If you want, next I can give you:
✅ A printable demo interrogation checklist
✅ Industry-specific demo tests (Healthcare / FinTech / AI)
✅ 5 killer demo questions that expose wrappers
✅ An investor demo scoring sheet
✅ Founder excuses decoder ring
✅ How to run a “hostile demo” politely 😄
Pick one.