We have a client who signed an AI pick-optimization vendor last fall. The first demo got applause from the whole room — orders auto-batched on a giant screen, pick paths optimizing like magic, the salesperson promising "two-week go-live, 99% accuracy." The owner was sold on the spot, and the contract got signed fast. Then go-live day arrived: the moment the system touched their real orders, it choked. SKU names were inconsistent, historical data had missing fields, and that "99%" turned out to have been measured on a cleaned, standardized dataset. The system sat in a corner for six months, gathering dust while the invoices kept coming.

If you're the buyer, here's the thing: a vendor demo is not a magic show. There's an iron rule for watching AI vendor demos — the first demo shows you the salesperson, the second demo shows you the engineering, the third demo shows you the contract. Skip any one of the three, and signing is gambling. This AI vendor demo evaluation checklist comes from sitting through more than a dozen AI vendor demos with our clients. Follow it, and at least you won't be regretting the signature.

Demo One: Watch the Function, Not the Fireworks

Demo one is the salesperson's home turf. His job is to make you go "wow." Your job is the opposite: cross out everything that made you go "wow," and check whether what's left is usable.

First, memorize the three most common demo lies and auto-translate them on the spot:

  • "AI-powered" → Translation: possibly a rules engine wearing a costume. Ask directly: "Which step in this workflow is actually handled by a machine learning model, and which steps are hard-coded rules?" If he can't answer, it's a rules engine.
  • "99% accuracy" → Translation: measured on lab data. Follow up: "What dataset was that 99% measured on? How large was the sample? Has it ever run on real orders in our industry?" Whether they dare run it on your data — that's demo two.
  • "Two-week go-live" → Translation: excludes data cleansing, excludes integration, excludes training. Ask: "Do those two weeks include the WMS integration and historical data cleanup? What happens to the timeline if our SKU master data is messy?"

One more move: ask the salesperson to change a parameter live. In a routing demo, switch the objective from "prioritize speed" to "prioritize full loads." In a wave-planning demo, change the wave cap from 50 to 30. A real system applies parameter changes in seconds. A recorded "demo" will expose itself on the spot — freezing, erroring, or the salesperson saying "we can prepare a customized version for you." The moment you hear "customized version," you know the feature doesn't exist yet.

Demo Two: Run It on Your Real Data

Demo two is the watershed — the one that eliminates 80% of vendors. It has exactly one requirement: run it on my data.

Prepare a "dirty data pack": three months of real, uncleaned orders (the dirtier the better), your SKU master data including the badly named ones, and one real peak-season day's wave volume. Have the vendor import it and run it in front of you. What are you watching for?

First, the import step. The moment real data goes in, the integrator's true colors show: mismatched fields, encoding garbage, missing-value errors — this is what your team will face every single day after go-live. If the vendor stumbles at the import step, do your own math on that promised "two-week go-live."

Second, explainability of results. You must be able to ask "why" about every optimization the AI suggests. If the system splits a batch of orders into two waves, you should be able to click through and see the reason: was it travel distance, order cutoff time, or labor scheduling? A black-box optimizer that can't explain itself will start a fight with your supervisor the first time an exception order appears — and the supervisor always wins. The system ends up benched.

Third, exception handling. Deliberately slip in a few bad orders: missing addresses, discontinued SKUs, absurd quantities. Watch whether the system flags them gracefully for human review, crashes outright, or — scariest of all — silently produces a wildly wrong recommendation with no warning. Everything going smoothly in a demo proves nothing. Not flipping over on exceptions proves everything.

After this round, the salesperson's "99%" usually drops to a number that sobers you up. Don't be disappointed — that's good news. Seeing the real number before signing beats seeing it after go-live, every time.

Demo Three: Pricing and Contract Are the Main Event

The first two demos are technical evaluation. The third is commercial negotiation. This is where many buyers get hurt, because the technical people leave after demo two and procurement faces the salesperson alone — information asymmetry, and you get fleeced.

First, understand how AI vendors price. Three models dominate today: per user, per order, and annual SaaS subscription. Each has its traps:

  • Per user: you add temp workers in peak season, the license bill grows with them. Clarify "concurrent users" versus "registered users."
  • Per order: costs scale with volume, which sounds fair — but ask about tiered pricing. "Above 100,000 orders a month, how much does the unit price drop?" If you don't ask, there is no discount.
  • Annual SaaS: simplest and most expensive. Clarify whether the annual fee includes upgrades, and whether major versions cost extra.

Then the hidden-fee trio, usually absent from the quote and appearing only in the contract: implementation fees (data cleansing, system integration — often half the software price), integration fees (per interface to your WMS/ERP), and training fees (per person-day). During negotiation, require the vendor to fold all three into the total quote. Anything not written in is assumed to be zero — in black and white.

Finally, three contract clauses that save lives. Don't sign if any one is missing:

  1. Acceptance criteria in the contract. The real metrics from demo two (accuracy, processing time) become acceptance terms, with the right to withhold payment or terminate if they're missed. A number promised verbally and absent from the contract equals no promise at all.
  2. Milestone-based payments. No more than 30% upfront, with the balance due only after go-live acceptance. Pay everything upfront and you'll spend the next year begging.
  3. Exit terms and data ownership. 30-day termination for convenience, your data stays yours, and the vendor may not train models on your data without your written consent. Skip this conversation now and you'll cry about it in two years when you want to switch vendors.

The Uncomfortable Truth: The Smoother the Demo, the Deeper the Pit

One last contrarian take: the slicker the demo, the more likely the post-signing crash. Because a slick demo means the data was prepped, the flow was rehearsed, and the exceptions were hidden. You're not buying those 45 minutes of theater — you're buying a system that works at 8 a.m. every morning for the next three years.

So the pros watch a demo for the "unslick" moments: how the salesperson reacts when stumped, what the error messages look like when dirty data goes in, how exception orders get handled. Those unslick moments are what the system really looks like.

Next time you're negotiating a contract, remember to ask these three questions: "Can you run it on my data?" "Can the acceptance criteria go into the contract?" "If we terminate early, what happens to my data?" A vendor who answers all three cleanly is worth talking to. One vague answer — thank you, goodbye.

By Susan, 3PL Sales Director