AI Pilot Site Selection: Why Your Best Warehouse Makes the Worst Pilot

Over the past two years, in interviews with more than twenty warehousing and retail companies that have run AI pilots, I have noticed a strikingly consistent pattern: the pilot almost always lands in the company's best facility — the flagship distribution center with the most stable network, the cleanest SKU master data, the most standardized processes, the most cooperative workforce, and the most supportive management. The logic seems impeccable: put the pilot where success is most likely, prove the concept, then replicate. Yet six to twelve months later, when these companies roll the solution out to their second and third sites, an equally consistent phenomenon appears: the impressive pilot numbers shrink dramatically, and some projects are quietly shelved during rollout. The pilot "succeeded," but the rollout failed. That paradox deserves a closer look.

The numbers: a gap between piloting and scaling

According to the MIT Project NANDA report "The GenAI Divide: State of AI in Business 2025," published in August 2025, roughly 95% of enterprise generative-AI pilots produced no measurable return on the profit-and-loss statement; only about 5% of organizations captured real value at scale. A note on methodology is warranted here: the 95% figure uses all surveyed organizations as the denominator, while among organizations that actually ran pilots, roughly one in four eventually reached production scale. Either way, the conclusion holds — a wide gap separates piloting from scaling, and most organizations never cross it. My observation is that the first step of that gap is often laid the moment the pilot site is chosen: site selection is itself a form of sampling, and piloting in your best warehouse is, in statistical terms, a biased sample.

Why the best warehouse makes the worst pilot sample

Three mechanisms explain the paradox.

First, endowment confusion. The flagship site's inherent advantages — clean data, standardized processes, stable networks, a cooperative team — make any solution look more effective than it is. Statisticians call this selection bias: you believe you are measuring the effect of AI, but you are actually measuring the combined effect of "a great warehouse plus AI," and attributing all of it to the AI. A meaningful share of the pilot ROI is paid for by the warehouse's endowment, not by the algorithm.

Second, resource distortion. A pilot period is an unusually pampered one: vendor engineers stationed on site, headquarters project managers following up daily, the warehouse's best people reassigned to support testing. These hidden inputs never appear in the pilot's cost denominator, yet they are real, and they genuinely inflate pilot performance. When resources are withdrawn during rollout, performance regresses — not because the AI broke, but because the greenhouse conditions disappeared.

Third, survivorship bias in the narrative. Pilots with beautiful numbers become case studies told and retold; mediocre pilots are quietly closed and never mentioned again. The "pilot ROI" figures circulating in the industry come from a filtered sample that systematically overstates reality — and in turn reinforces the seemingly obvious instinct to pilot in the best warehouse.

Consider the first case. In the second half of 2024, a 3PL launched an AI vision-based audit pilot in its flagship warehouse near Suzhou. After three months, audit accuracy rose from 99.2% to 99.6%, exception interceptions nearly doubled, and the project team estimated annualized labor savings of roughly RMB 400,000. The pilot was rated an "above-expectations success." In early 2025, the same equipment and the same algorithm version were rolled out to regional warehouses in Wuxi and Hefei — accuracy improved by less than 0.1 percentage points, essentially noise. The post-mortem found the key difference: the flagship's audit stations had been renovated in 2023, with even lighting and fully standardized pallets and totes, while the regional sites had aging lighting and a jumbled mix of carton sizes. What the vision model had "learned" during the pilot was the visual signature of a standardized environment. In other words, the pilot measured not the algorithm's ability to generalize, but the friendliness of the pilot environment.

The second case comes from the first quarter of 2025, when a US retail chain piloted demand-forecasting AI in its flagship distribution center near Dallas, serving about 120 stores, with 98% SKU master-data completeness. Six months later, forecast error (MAPE) had fallen by about 4 percentage points and stockout rates by 1.2 points; the verdict was "success, recommend rollout." Replicated at two Midwestern DCs — where master-data completeness was barely above 80% and promotion records were patchy — the model delivered only about 30% of the pilot's gains. The team then re-ran the pilot in a deliberately "average" warehouse: data gaps, process exceptions, lukewarm cooperation. The numbers were less flattering than the first pilot, but the data-governance checklist and rollout playbook built from that second pilot landed successfully in all four subsequent warehouses on the first attempt. The project lead put it plainly: the first pilot proved the AI could work under ideal conditions; the second proved it could work under our real conditions. Management only ever needed the second proof.

Here is the uncomfortable inference: a pilot whose ROI beats expectations deserves extra scrutiny, not celebration, because the outperformance may owe less to the AI and more to the pilot site being unusually good. A pilot run in a below-average warehouse, with numbers 20-30% uglier, measures the solution's floor; a pilot run in the flagship measures only its ceiling. What management actually wants was never "pilot success" but "rollout success" — and the site-selection logic for those two goals points in opposite directions: the first seeks safety, the second seeks truth.

Three lines for pilot site selection

One more element is worth adding: a control. If possible, designate a "shadow control" — a comparable site in the same region that does not get the AI but records baseline data over the same period — or run category- or time-based comparisons within the same warehouse. A pilot ROI without a control group is, strictly speaking, an unfalsifiable number; its persuasive power exists only inside the slide deck.

A shift already underway

As enterprise AI spending moves from the experimentation phase into the scaling phase, pilot methodology is changing. The early logic was "get it running first," with site selection optimized for safety and speed. The emerging logic is "make it replicable first," with site selection optimized for truth and representativeness. My expectation: by around 2027, pilot-site representativeness assessment will become a standard step in enterprise AI governance — not because the industry suddenly became rigorous, but because enough companies have now paid tuition. Whoever fixes site selection first will pay the lowest tuition in the rollout phase.

Finally, one question to take with you. If an AI solution only produces beautiful numbers in your best warehouse and falters everywhere else, should we ask "does the AI work" — or "are our warehouses worthy of this AI"? And further: when the choice of pilot site can largely determine the pilot's outcome, how many of the industry's celebrated "AI success stories" of the past two years measured the technology, and how many merely measured good taste in site selection? There is no standard answer. But every company about to launch a pilot would do well to ask itself that question once — before signing.

By Dr. Chen, Supply Chain Researcher