Our little warehouse cannot afford to burn money like that, so when it comes to AI POC acceptance metrics, I check everything twice.
Here is our situation: a 3,000-square-meter warehouse, 12 people, more than 800 orders a day. In early 2025 a vendor came to pitch visual quality inspection — "99.5% accuracy, payback in three months." The demo video was beautiful: boxes gliding down the conveyor, green boxes popping up on screen one after another. I almost signed on the spot. Then I asked one question: were the goods in the demo picked by you? The salesperson smiled and changed the subject. In that moment I knew this money could not be spent on a demo.
AI POC acceptance metrics are not about how good the demo looks. It is about how the system performs in a real warehouse. The demo room has bright lights, neat boxes, and full-bar Wi-Fi. A real warehouse has half the lights flickering, crooked boxes, and Wi-Fi dead zones everywhere. An accuracy rate of 99% in the demo can easily drop to 85% in real operation — and at 85%, the system is worse than Lao Zhang's eyes.
The Demo Looks Great; the Real Warehouse Is the True AI POC Acceptance Test
Here is the first story (illustrative scenario). In March 2025 we trialed a visual inspection vendor. In the demo it used standard cartons with crisp printed labels and claimed 99% accuracy. In our warehouse, labels were half-covered by tape, cartons were crushed, and barcodes were dusty. First-week accuracy came in at 87% (illustrative example). A 13% error rate means 13 out of every 100 orders need manual re-checking — more manual work than our old fully-manual inspection. That AI was, effectively, bought for nothing.
The gap between demo and reality comes down to three things. First, the samples: demos use the "good students," while real warehouses are full of edge cases — damage, stains, glare, wrinkles, everything. Second, speed: a demo scanning 20 items a minute slows down fast when the conveyor speeds up and boxes crowd together. Third, endurance: a demo runs for 20 minutes, but a real warehouse runs 10 hours, and overheating, slowdowns, and dropped connections in hour six never show up in a demo.
So my down-to-earth method is this: before acceptance, make the vendor run two weeks on our real order volume, using our hardest goods. Do not use their test set — use our worst boxes. The numbers from that run are the only ones that belong in a contract.
Five AI POC Acceptance Metrics to Put in the Contract
Here is the counterintuitive part first: an AI with 99% accuracy can still lose you money. If it protects its accuracy by pushing every doubtful case to a human, and the manual intervention rate hits 40%, you have paid for AI and kept the manual work. Acceptance cannot rest on a single number. All five metrics have to pass together.
| Metric | How to measure it (verified by live test) | Fail line |
|---|---|---|
| Accuracy / error rate | Count false positives and false negatives separately on real orders | False positives above 2%, or missed defects above 0.5% |
| Speed and throughput | Real throughput over 4 continuous hours, excluding setup downtime | Below 90% of promised speed |
| Stability and uptime | 8 hours of continuous running without a crash or restart | More than 1 crash in 8 hours |
| Manual intervention rate | Share of orders needing human review or takeover | Above 5% |
| True ROI payback | Labor saved minus new costs, calculated from live test data | Payback longer than 18 months |
Accuracy must be split into false positives and false negatives. A false positive flags good goods as bad and wastes labor; a missed defect lets bad goods through and costs customers and compensation claims. Both numbers go in the contract. Any vendor reporting a single "accuracy" figure is playing games.
Speed must be measured over four continuous hours at real volume. A ten-minute sprint in a demo does not count — what matters is holding speed through a four-hour peak. Stability is even more concrete: run eight hours straight, and more than one crash means fail. AI is equipment, and equipment gets tested like equipment.
The manual intervention rate is the number I care about most. In a small warehouse, 5% and 15% are two different businesses. At 5%, one person checks in part-time; at 15%, you need a dedicated person watching the system, and the labor you saved comes straight back. ROI works the same way: labor saved, minus electricity, maintenance, consumables, and the wages of the reviewer you still need. Net savings divided by investment is the real payback period. Vendors often count only the savings and skip the costs — so redo the math yourself.
The Number Games Vendors Play — and the Traps I Have Stepped In
Here is the second story (illustrative scenario). In May 2025 another vendor pitched demand forecasting. The demo showed 92% forecast accuracy, which sounded great. On closer questioning, the 92% covered only bestselling items; long-tail products were excluded entirely. Long-tail items make up 60% of our warehouse, so that 92% was nearly useless to us. Another vendor claimed "5,000 orders a day" — with fine print saying "lab conditions." In our real network and lighting, throughput dropped 30%.
The usual vendor number games are: report the prettiest subset, report lab conditions, credit the AI for work humans actually did, and count revenue in ROI while ignoring costs. The fix is simple and stubborn: every number is verified by live test, and every figure in the contract carries the phrase "based on two consecutive weeks of live testing in our warehouse." Any number without a source gets treated as "subject to live test" — I do not argue about it, I just measure it.
There is also the "free pilot" trick. It sounds free, but when the pilot ends, the data formats, interfaces, and training are all locked to that vendor, and switching means starting over. Free is the most expensive price there is, and that applies to AI pilots too. Before any pilot, ask one question: when the pilot ends, can I export all my data in full? If not, I will not do it even for free.
If Acceptance Fails: Write the Exit Clause First
The most important part of the contract is not the AI POC acceptance metrics — it is what happens when acceptance fails. My version has three clauses, and none can be skipped.
First, staged payment: 30% on signing, 40% when live testing passes, and the final 30% after three months of stable operation. Whoever holds the money holds the leverage. With a contract paid in full upfront, you have almost no negotiating power at acceptance time.
Second, a fix window and an exit right: if acceptance fails, the vendor gets 30 days to fix it, and if the fix fails, we can terminate the contract and recover 70% of what was paid (illustrative example; actual terms follow the signed contract). Cap the number of fix attempts at two — do not let "just one more tweak" drag on forever. Three months of tweaking and peak season is already gone.
Third, data ownership: all labeling data and tuning records produced during the pilot belong to us, and the vendor hands everything over in full when they leave. This clause prevents lock-in. Without it, switching vendors means starting from zero.
A small warehouse has its own way of surviving. Our method is rough, but it works: do not trust the demo, trust the live test; do not trust one number, trust five metrics; do not sign a contract without an exit clause. Whether AI can actually save a small warehouse money is decided by the five numbers on the acceptance sheet — not by the numbers in a sales pitch. Tested this way, at least one thing is guaranteed: the money is spent with eyes open, and not wasted.
By Xiao Ma, Small Warehouse Owner







