Here's the verdict up front: every AI vision project that hits 99% accuracy at acceptance will slide into the low 90s within six months. That part doesn't surprise me. What surprises me is that nine out of ten contracts I've reviewed never say who owns that slide.
In the fall of 2024, a 120,000-square-foot 3PL warehouse in Riverside asked me to do a post-mortem. Earlier that year they'd deployed a vision-based cycle counting system: fixed cameras down the rack aisles, a model that counted pallets and recognized SKUs every night. Acceptance accuracy was 99.2%, and the boss praised the IT team at the company meeting. By Q4, count variances kept widening. A manual reaudit found the model was confusing a repackaged SKU for a different one — three straight months of book inventory disagreeing with physical stock, right through peak season replenishment. The root causes piled up fast: 40-plus new SKUs launched over the summer, all with new packaging colors; the night shift had dimmed the aisle lights two notches to save power; new operators were stacking pallets crooked in ways the training data had never seen. Not one of these was "the model broke." Every one of them was "the world changed."
That's model drift. Go-live day is the most accurate day of an AI system's life. It only gets less accurate from there — the only question is how fast. The people who do selection talk about go-live. The people who do operations talk about the five years after.
Drift Is a Season, Not a Bug
Drift comes in three flavors. First is data drift: what the cameras see has changed — new SKUs, new packaging, dimmer lights, dust on the lens. Second is concept drift: the business rules changed. Mixed pallets used to be forbidden; now peak season quietly tolerates two SKUs per pallet, and the model falls apart. Third, and sneakiest, is operational drift: people's behavior changed. Same SOP, but a rookie stacks pallets nothing like a twenty-year veteran, and the model only ever learned the veteran's technique.
You don't catch drift by feel. You catch it with machinery. The standing rules I give clients are four. Run two weeks of shadow mode after go-live, reconciling model output against manual counts daily — the gap is visible on day one. Then do fixed weekly spot checks, 200 pallets minimum, and convert the results straight into an accuracy number. Set a red line — say, SKU-level accuracy below 97% for two consecutive weeks — that pages a named human being. And do a full reaudit before and after every peak season, because seasonal changeover is drift's favorite hunting ground. Miss any one of these four and drift gets discovered three months late, like in Riverside, when the books have already been wrong for a quarter.
Where the Money Actually Goes
Most people think maintenance means "call the vendor when the model misbehaves." Wrong. Ongoing operations spend money in three places, and each one needs an owner.
First is data recirculation. Every retraining cycle needs hundreds to a thousand-plus freshly labeled images: somebody has to photograph the problem cases on the floor, somebody has to label them, somebody has to QC the labels. Labeling is the thing everyone plans to do "when there's time," which is why three months pass without a usable dataset. Second is the retraining itself. Renting cloud GPUs for a run takes anywhere from a few hours to a couple of days; the compute bill is a rounding error against the project total. The real cost is the days you wait, when the business runs on manual workarounds — during peak, that hurts. Third is the person who watches the alerts. Whose job description says "reads the accuracy report every week"? Whose phone gets the red-line page? If it's nobody's job, it's nobody's job — I've watched alerts sit unread in a group chat for three days more than once.
My call is blunt: reserve 15 to 20 percent of the project's first-year spend for ongoing operations. That's the common thread across every warehouse I've seen that didn't blow up. Spend less than that, and the day the vendor's "one year of free maintenance" expires is the day accuracy starts its free fall. Whatever you saved comes back as count variances and overtime, with interest.
The Lines Your Contract Needs Before Go-Live
When you're negotiating with the vendor, don't just haggle over price and acceptance accuracy. Get these lines into the body of the contract — and if a vendor won't agree, find another vendor.
First, how accuracy is measured: SKU level or pallet level, sampled or full-population. The same system can be 99% one way and 95% the other, so align on the yardstick before anything else or you'll be arguing past each other for months. Second, what happens below the red line: response within how many hours, a retraining plan within how many days, back above the line within how many days after retraining. Every tier needs a clock. Third, who owns the data: recirculated labeled data belongs to you. The vendor may use it to improve the model, but not to build somebody else's project — skip this clause now, litigate it in two years. Fourth, a retraining floor: at least quarterly, plus one extra before peak season, with the dates written down. Fifth, who supplies the labeling labor: your people photographing and QC-ing on site, or theirs, and who pays.
Everything not in the contract becomes a negotiation later. The most expensive argument I've witnessed was two months of fighting over the definition of "accuracy" — while the model kept miscounting and the books kept drifting. The arbitration bill alone would have paid for two retraining cycles.
The Checklist
Print this, tape it to the IT office wall. It beats any "AI strategy" deck, and you can start tomorrow:
- Accuracy definition in the contract (SKU vs. pallet level, sampled vs. full)
- Weekly spot-check volume locked in (200 pallets minimum recommended)
- Red-line threshold and a named first responder (name and phone number, not a group alias)
- Where recirculated data lives, and who photographs, labels, and QCs
- Retraining calendar set (quarterly, plus one before peak)
- Vendor response SLA written down (hours to respond, days to deliver the retrained model)
- A named owner for syncing lighting, packaging, and SKU changes to the AI lead
- Shadow-mode reconciliation running from day one of go-live
