In September 2025, the general manager of a 3PL warehouse in eastern China called me about a vision-recognition pilot. The vendor's demo had hit 99.6% recognition accuracy, the contract said "accuracy no lower than 99%," and the acceptance form was already signed. In the first week of live operation, damaged labels, reflective tape and crooked barcodes flooded in, and accuracy fell to roughly 91%. The rework station needed two extra labor hours a day to clean up the mess. He asked me whose fault it was. My answer was that it was the contract's fault. If AI POC acceptance criteria consist of a single line reading "no lower than 99%," with no definition of the test set, no sampling method and no measurement protocol, that line has no binding force at the acceptance meeting. A vendor can pick 2,000 clean photos of standard cartons and the number still holds.
This is not an isolated case. Judging by the delivery terms several leading integrators publish, and by the dozen or so warehouse AI projects I have tracked over the past two years, the most common way a POC dies is not that the technology fails. It is that the acceptance standard was written too loosely.
Turning AI POC Acceptance Criteria Into Four Measurable Numbers
A function that works in the demo has to survive three transitions before it works in production: the data distribution changes, the operating rhythm changes, and the exceptions show up. If the contract defines success only in the demo environment, all three transitions become the buyer's cost.
| Category | Headline metric | How it is measured | Common trap |
|---|---|---|---|
| Accuracy | Recognition or prediction accuracy, error rate | Continuous sampling from live flow, at least 5,000 units, stratified by product category | Testing only the demo set, or reporting overall accuracy without the worst category |
| Speed | Processing time per unit, P95 system response | Measured continuously through peak hours, using the 95th percentile, not the average | Reporting averages only, which hides the long tail of delays |
| Stability | Continuous run time without manual intervention, time to recover from failure | 72 hours of continuous operation, logging every stoppage and recovery | Testing only an eight-hour day shift, never the night shift or a shift change |
| Cost | Cost per unit processed, labor savings converted to money | Calculated from actual volume and labor hours during the acceptance period | Counting only the software license, ignoring rework, maintenance and exception handling |
Accuracy is where the wording most often goes wrong. The same "99%" can describe two different worlds depending on whether it was measured on a vendor-curated demo set or on live flow. The contract should state that the test set comes from the buyer's real freight, that sampling covers all product categories and exception types, and that results are judged category by category, with the worst category scored separately rather than hidden inside an overall average. Speed should be written as a P95 value, not an average, because a warehouse never suffers at the average: it suffers when the worst 5% of orders jam the line. Stability should be written as continuous run time, and 72 hours is a practical yardstick, covering a day shift, a night shift and one handover.
Cost is the metric most easily reduced to an empty sentence. "Expected to save 30% of labor" cannot be accepted, because it has no baseline. The workable version is to fix the baseline first — the actual cost per unit and labor hours for that process in the two weeks before the POC — and then compare it against the measured values during acceptance. Rework, exception handling and maintenance labor have to be included, or the hours the vendor saves will be eaten up again at the rework station.
Passing on Demo Data, Failing on Real Flow
A second case comes from an e-commerce warehouse in Guangdong at the end of 2024. The vendor's AMR scheduling algorithm promised a 25% lift in picking efficiency, and acceptance was measured on a single wave in an idle period. The numbers looked excellent. When the peak-season simulation ran, the lift fell to 7%, and after 72 hours of continuous operation the system hit a scheduling deadlock that took 40 minutes to restart. The contract said nothing about continuous run time, recovery time, or performance under peak concurrency. The POC was marked "passed," and the problems surfaced in bulk three months after go-live, when the rework cost more than the POC itself.
Both cases point to the same cause. There is a systematic gap between the POC test environment and the production environment, and the size of that gap depends on whose data was used, at what pace it was tested, and for how long. Demo data is the vendor's comfort zone; real data is the buyer's daily reality. My recommendation is to fix three things in the contract: first, test data must come from the buyer's real freight, sampled to cover peak-season wave patterns and exception types; second, acceptance measurement must cover at least one full peak operating period, never an idle window alone; third, the acceptance period must run for at least two weeks, including one shift handover and one complete exception-handling cycle.
There is a counterintuitive point here: the higher you set the accuracy target, the more easily a POC gets distorted. A 99.9% target sounds rigorous, but in practice it often pushes the vendor to overfit the demo set, spending their energy polishing the test set to perfection rather than making the system more robust. A useful acceptance target is not a single impressive number but a tiered set of metrics: overall accuracy, worst-category accuracy, and exception-type coverage, all three of which must pass together. The trade-off is clear. A looser-looking but structurally complete set of metrics buys you credibility on real freight; a single very high number usually buys you a good demo.
If Acceptance Fails, the Contract Needs an Exit
Once the metrics are clear, the contract still has to say what happens when they are not met. Many POC contracts have only one sentence here — "if acceptance fails, both parties will negotiate" — which in practice means a quarrel. A more durable structure is a tiered exit: on the first failed acceptance, the vendor gets one free remediation within 30 days, with the scope and cost cap written into the contract; if it fails a second time, the buyer may terminate, recover the unspent implementation fees, and settle delivered hardware at depreciated value, with data and model ownership fixed at signing.
The arithmetic shows what that clause is worth. Take a POC that costs 400,000 yuan with a full rollout budget of 4 million. If the acceptance standard is vague and the system is declared a pass, only to fall short during rollout, rework and remediation can easily consume another million yuan, plus three months of lost window. Against that, two pages of acceptance terms cost almost nothing. The other side of the same calculation is that a POC is, in essence, a small payment for information — the answer to whether this solution actually works in my warehouse. If the acceptance standard is written so the vendor cannot lose, the money buys not information but a ticket to rollout.
A successful POC also needs a handover clause for the move to production. Three things belong in the contract: first, the POC acceptance data and test set are kept as the go-live baseline, and production metrics must not fall below the POC acceptance values; second, rollout proceeds in stages, each with a checkpoint, and a missed checkpoint stops the rollout at that stage; third, the vendor's on-site support period and response times are written into the rollout terms rather than left as verbal promises. The most dangerous break between POC and go-live is the moment acceptance ends and the vendor's best people walk out, leaving a system nobody is accountable for.
The judgment, by now, is fairly clear: POC acceptance is not a technical review but a pricing of risk. The metrics written into the contract decide who bears the uncertainty of real freight; the exit clause decides who has the right to stop when that uncertainty exceeds expectations. Over the next two or three years, warehouse AI projects will only multiply, and vendor demos will only get slicker. The way to tell whether a vendor is serious is shifting from watching the demo to watching whether they will put the AI POC acceptance criteria in the contract, and whether those metrics survive contact with real freight. As for your next POC, who gets to set the numbers on the acceptance form is a question worth settling before the contract is signed, not after.
By Dr. Chen, Supply Chain Researcher







