Honestly? The words "AI vendor sales demo" now trigger an involuntary eye-roll from me. Nothing against AI — I just got burned by demos a couple of years ago, and the tuition was painful. So here's what that tuition bought: the 5 questions I now ask at every single demo. Save them before your next one. They might save you.
The Uncomfortable Truth First: Every Demo Is Rehearsed
Here is the counterintuitive bit: what you see in a demo is not the product's everyday state. It is the highlight reel. The sample data is cleaned, the SKUs are cherry-picked, the lighting, the network, the workflow — all rehearsed. It is like a first date. Everyone brings their best self. The problem is that warehouse reality means reflective packaging, wrinkled barcodes, crooked pallets, and peak-season waves. None of that shows up in the demo.
Early 2025, I watched a demo from a vision-based sorting vendor. It was buttery smooth on site: camera scans, SKU info and quantities all correct, the robotic arm never missing. I nearly signed a letter of intent on the spot. Back home I ran three months of my own warehouse's real order data through it — reflective packaging it could not read, wrinkled labels it misread, mixed-SKU pallets it simply gave up on. What looked like a hundred percent in the demo fell apart so badly in my real operation it was unusable. Since then I have one iron rule: never make any decision at the demo itself.
The 5 Questions — Skip None of Them
These five are now my standard kit for any AI vendor demo. Ask them one by one, and only talk pricing after.
1. Can you run it on MY real data?
How to ask: hand over de-identified real data — three months of orders, SKU master data, problem-case logs — and have them run it live. What to watch for: vendors who say yes without flinching at least trust their own generalization. Vendors who stall and insist "our sample data is more standard" are performers who only dare to play on their home stage. Red-flag answer: "Our model performs best on standard data." Translation: it cannot handle your real data.
2. Can we test my nastiest scenario?
Every warehouse has its own personal hell. Mine is reflective poly-bags plus wrinkled shipping labels. At the demo, say it out loud: skip the easy stuff, run this. AI products do not fear the average case; they fear the long tail. Remember: the slickest-looking part of the demo is usually the first thing to break after go-live — because it is the part they rehearsed the most.
3. Which project did you mess up?
My favorite question. Ask for a failure story and watch whether they dare to tell one. A vendor who can say "after go-live at one client our recognition rate dropped, and we spent two weeks adding lighting and a second verification pass" has actually shipped real projects. One who stammers "we don't really have failure cases yet" is either lying or too green. Pick neither.
4. Who maintains this thing after go-live?
The demo sells capability; maintenance decides lifespan. You must pin down: how often is the model retrained, and by whom? Who labels my new SKUs when they launch? What is the response SLA when something breaks? Plenty of AI vendors sell "turnkey" — then hand over the key, walk away, and leave you wrestling the model alone.
5. Exactly what am I paying for?
Per API call, per warehouse, per year, or per successfully recognized unit? How do price-increase clauses read? Who owns my data — can it be used to train someone else's model? Ask pricing last, but ask it thoroughly. Any fee left vague in the demo will become a thorn in the contract.
Do the Math: Trust No Number in the Demo
Time for data. Vendors typically quote two kinds of numbers in demos: accuracy and ROI. My policy: accuracy only counts if it comes from running my data; ROI only counts after subtracting maintenance labor and labeling costs. Here is the simplified version of the scale in my head:
| At the demo | My acceptance standard | |
|---|---|---|
| Data | Vendor's sample data | Three months of my real, de-identified data |
| Scenario | Cherry-picked easy SKUs | My personal-hell scenario is mandatory |
| Accuracy | A number quoted on stage | Measured on my data, long tail included |
| Maintenance | "Basically maintenance-free" | Retraining, labeling, and SLA pinned down |
| Price | First-year promo price | Three-year total cost of ownership |
And a decision framework: score each of the 5 questions 0–2 (0 = cannot answer, 1 = shaky, 2 = crisp answer with evidence). Talk business only at 8 or above; pass outright below 6. I have used this scorecard several times now, and it filtered out two "demo champions, deployment disasters."
Three pitfall warnings: first, sign nothing at the demo — give it 48 cool-headed hours. Second, get every key metric promised in the demo written into the contract as an appendix, in black and white. Third, structure the POC as pay-for-performance, with the right to walk away if promised metrics are missed. A vendor willing to accept all three is usually the real deal.
Learn to decode the three most common demo phrases while you are at it. "Our algorithm is proprietary" — proprietary does not mean proven; what matters is whether it has actually shipped in your industry. "99% accuracy" — an accuracy number without a denominator is theater; 99% on sample data versus your personal-hell SKUs are two completely different sports. "Live in two weeks" — going live and running well are two different things; the first three months after launch are the real exam. The more demos you watch, the clearer the pattern gets: demos sell certainty, warehouses run on uncertainty.
Side note: I shared this scorecard with two friends running 3PL warehouses — they used it on WMS vendor demos and it worked just as well. The logic transfers: any vendor afraid to let you verify with real data earns a question mark by default.
"Run it on my data" — this move? Personally tested. Works. Twice it filtered out demo champions for me, and once it talked a first-year quote down substantially — when vendors realize you actually know how to inspect the goods, their pricing gets a lot more honest. Next demo, print these 5 questions and bring them with you.
By Ah K, E-commerce Seller (5 yrs)
