Last fall, I was advising a 3PL warehouse in the Inland Empire — about 120,000 square feet — that had just deployed an AI model for replenishment forecasting. The acceptance form was signed. The vendor's team left in week three. On day 96, the warehouse manager called me: "Lao Mi, I think the system's broken. Its replenishment suggestions are getting crazier every week."
I asked three questions. What was the model's forecast accuracy last week? He didn't know. Where do the alert emails go? He went to check — and found they were still going to the mailbox the vendor's project team had set up. That team had been disbanded for two months. The alerts had been landing in an inbox nobody opened. Third question: who set the drift thresholds? Silence on the other end of the line.
This isn't an isolated story. I've watched several AI projects die the same way over the last couple of years — not during selection, not during development, but in the three months after go-live. When you accept a piece of equipment, a forklift doesn't change lanes by itself. AI is different. It eats new data every day, and when the data changes, the model changes with it. With nobody watching, the "AI system" you own three months later is no longer the one you signed off on.
Go-Live Day Is the Riskiest Day of the Project
Here's a counterintuitive take: for an AI project, the acceptance signature marks the beginning, not the end.
Acceptance testing happens under static conditions. The data was cleaned by the vendor in advance, the test set was curated, and the seasonal mix was hand-picked. Of course the vendor wants the acceptance metrics to look good — the final payment is on the line. That's not dishonesty; it's human nature.
But once real data starts flowing in, everything shifts. New customers bring new order patterns. The SKU mix changes. Packaging specs get swapped. Peak and off-peak seasons flip. This is the "dirty data" the model never saw during training. I watched one warehouse go from 92% forecast accuracy at acceptance to 78% two months later. The root cause was almost funny: the training data had no Q4 peak-season SKU combinations, so the model started guessing wildly the moment peak season hit.
So my verdict is blunt: the acceptance criteria must include a 90-day retest clause. Sign twice — once at go-live to confirm the system is running, and again at day 90 to confirm the model still holds up against real data. If it fails the second test, the final payment stays on the table and the vendor comes back to tune it. One clause in the contract beats ten extra metrics on acceptance day.
A lot of buyers are afraid to ask for this. They worry the vendor will take it as distrust. I see it the opposite way: a vendor willing to sign that clause is one that actually believes in its model.
Models Don't Die Suddenly. They Get Dumber Slowly.
This is the biggest difference between AI operations and traditional IT operations. A server crash is sudden — the alarm goes off and everyone knows. Model drift is a chronic illness. A little worse today, a little worse tomorrow, and three months later you realize it has quietly become useless.
Drift comes in two flavors, in plain terms. First, the data changed: you used to ship 8,000 orders a day, now it's 12,000. Customer A used to be 60% of volume; now Customer B dominates. The model learned the old data's patterns, and the new data doesn't follow them. Second, the relationships changed: "weekends are busy" used to hold true, then customers shifted their ordering habits and weekends went quiet. The data itself looks the same, but what the data means has moved.
The scary part isn't drift itself — it's that nobody is watching. At that Inland Empire warehouse, the system had been sending alerts for weeks. The drift metrics were already over threshold. But the alerts went to an inbox nobody read, which is the same as no alerts at all. That's the operations vacuum I'm talking about: the system is talking, and nobody is listening.
How do you fill it? The responsibility chart has to be set before acceptance, and it has to name actual people — "the client is responsible" doesn't count. I recommend three layers.
Warehouse operations watches the business metrics daily. Nobody feels forecast quality faster than operations — when replenishment suggestions are nonsense, the pickers complain first. Give them one simple rule: if suggested quantities deviate from reality beyond a fixed ratio for three straight days, escalate. No algorithm knowledge needed. Just read the numbers.
IT — or the vendor — watches the model metrics weekly. Drift scores, feature distributions, prediction confidence. This needs someone who knows what they're looking at. If you don't have that person in-house, buy remote model support from the vendor in the contract, quarterly billing is fine. Don't cheap out here. Skip it, and the hundreds of thousands you already spent can go down the drain.
Management reviews the trend monthly. Is the accuracy curve going up or down? What did retraining cost, and was it worth it? Thirty minutes is enough, but the meeting has to happen. Otherwise you discover at the quarterly review that the model has been broken for three months. Too late.
Who sets the drift thresholds? My answer: the vendor proposes, the buyer decides. The vendor knows the model and what counts as normal fluctuation. The buyer knows the business and what error level is unacceptable. Set them together, write them into the SOP, and put it on the wall.
A Handover That Isn't in the Contract Didn't Happen
This is the deepest pit I've fallen into. At one warehouse, an AI quality-inspection project, the contract said "the vendor will provide training." Before acceptance, the vendor did provide training — a two-hour slide deck. Three months later the model acted up, and the operations engineer didn't even know where the feature pipeline lived. Which table the data came from, which server held the model files — three questions, zero answers. He called the vendor. The project team was gone. A support rep said they could "open a paid support ticket."
So now I make buyers check off a handover list item by item before signing acceptance. Four items. Miss one, don't sign.
First, model documentation. What the feature list is, which version of the training data was used, what assumptions the modeling relied on — things like "assumes SKU count stays constant" or "assumes order structure is stable." Those assumptions are your future drift warning lines. The moment an assumption breaks, the model needs retraining. The docs don't need to be fancy. They need to be readable.
Second, alert configuration. Who gets the alerts, who is cc'd, how often they're reviewed, who escalates. Write out every email address, and send a test alert on acceptance day to confirm it arrives. Don't laugh — I've seen a project where the alert address was misspelled and nobody found out until after go-live.
Third, the retraining process. Who triggers retraining, what the trigger conditions are, who executes it, and who pays. Retraining isn't free — data prep, compute, and validation all cost money. If you don't settle this upfront, both sides start finger-pointing when something breaks. I watched one dispute drag on for four months. The model just sat there rotting for four months.
Fourth, the rollback plan. When the model misbehaves, how do you switch back to the old rules engine with one flip? Where is the switch, who has permission to flip it, and how does the business keep running after the switch? This one gets skipped most often, but when things go wrong, it's the one that saves you.
The contract needs two more clauses: remote vendor support for 6 to 12 months after the team leaves, with response times spelled out; and data ownership — the training data, the model files, the feature pipeline — the buyer gets a complete copy of all of it. The vendor can leave. The assets stay.
Here's the bottom line. An AI project differs from buying equipment in one fundamental way: equipment is a one-time delivery, but AI is something that keeps evolving. What you're accepting isn't a machine. It's a living thing that needs continuous feeding. If the contract doesn't say who feeds it, it's only a matter of time before it starves.
One question to leave you with: if your AI project goes live tomorrow, can you name whose inbox the alert emails go to on day 90 — and who is responsible for opening them? If not, don't sign the acceptance form yet.
