It was 6:50 on a Tuesday morning this past April, still half dark outside.
Night shift was clocking out, day shift hadn't fully shown up. My phone rang — the QA manager from one of our suppliers. No small talk: one lot of frozen beef patties, potential metal contamination on the line, lot L-20260411-A, shipped to us in two deliveries last week. They needed an answer in two hours: how much of that lot was still in our building, how much had shipped, and to whom.
Honestly? My stomach dropped. Not because I was afraid of blame. Because I knew our lot tracking couldn't survive that question.
How the problem surfaced
I pulled our QA lead and receiving supervisor into the office. We looked up the lot in the WMS. Three problems, right there on the screen.
First, the lot arrived in two deliveries. The second one — 18 pallets — receiving took it in as full pallets and put it straight up high. The pallet tag had the location. No lot number. The lot code was only printed on the cases, and the cases were buried in the middle of the pallet. You'd have to climb up to see it.
Second, outbound was worse. Six pallets had been broken down for piece picking, mixed into wave batches for three different customers. The "lot number" column in the pick records? Blank. All blank. Turns out two veteran leads had a habit: scanning lot codes during break-bulk picking was a hassle, so they skipped it. The new hires learned fast — learned the wrong thing.
Third, two pallets had been transferred last week to our San Bernardino building. Transfer order printed, truck gone, lot information gone with it. The crew over there picked up the phone just as confused as we were.
That day it took us 5 hours and 40 minutes to account for the lot. And we still missed one pallet — found it later in the returns staging area, buried under three layers of other freight. The customer didn't get an answer in two hours. They called my boss instead.
Driving home that night, one thought kept looping: if this had been the FDA knocking instead of our supplier, we'd have been done.
How we figured it out
After that, I made a rule: one recall drill every quarter. Random lot, on the clock, two hours.
Our first drill, we started the obvious way — in the system. QA clicked around for 40 minutes. The system data didn't match the floor at all. In the end it was people climbing racking, reading case labels one by one. Three-plus hours. The drill report looked great. The floor was a mess.
Second drill, we flipped the order: lock down first, check the system second, verify by hand last. Got it to 2 hours 10 minutes. Still over.
Here's the counterintuitive thing we learned in the debrief: 90% of drill failures start at receiving, not in the system. The system could do it. The WMS had a lot-number field. Nobody filled it in properly. Receiving rushed full-pallet intakes without checking. Putaway mixed lots on the same pallet. Returns got restocked with the old label torn off. Small stuff, day to day. Landmines, when it counts.
One more lesson: never announce the drill in advance. We ran one announced drill once — told everyone a week ahead it'd be Tuesday morning. Passed in 38 minutes. Everybody performed beautifully. Then I ran a surprise one. Same people, same building: 2 hours 40 minutes. An announced drill is theater.
The two-hour playbook: this is how we run it now
Here's our fixed playbook, in four phases, clock enforced:
0–15 minutes: LOCK. The recall team assembles — me, QA, the receiving lead, the shipping lead. Four people, on-site within 10 minutes of the call. The hold area gets cleared — we've got a fixed zone in the southeast corner, red floor lines, no normal freight allowed there, ever. Every suspect location gets frozen in the WMS. Locked. Nobody touches anything.
15–45 minutes: TRACE. Pull one-up-one-down in the system: upstream — which supplier, which receiving ticket, which truck; downstream — which customers, which outbound orders, which transfers. Check returns too. Returned product may have shipped back out.
45–90 minutes: VERIFY. Pairs on the floor, scanning every pallet, photos for the record. For broken-down pallets, work backward through the wave sheets. This is the grind, and it's non-negotiable — the system says location A, the floor says location B, the floor wins.
90–120 minutes: REPORT. The list goes out: lot number, quantities, what's in the building, what's in transit, what shipped to whom, with contacts. Written, to the customer and the supplier's QA, management cc'd. Phone calls don't count. Paper counts.
A few hard rules we bought with those lessons:
Receiving: lot number is mandatory. The WMS field is required — no scan, no receipt confirmation. One lot per pallet. Mixed pallets get rejected, and the receiving lead signs for it.
Transfers: the lot number rides with the truck. Cross-building transfer paperwork must print the lot code. Driver signs, receiving building scans to confirm. Break the chain once, answer for it.
Returns: treat them as new receipts. Old label off, new label on, rescan. The returns area is where we lost that pallet last time. It's watched now.
Drills: quarterly, unannounced. The target lot sits sealed in an envelope until drill morning. Every drill ends with a debrief, problems go on the whiteboard, and the next drill starts by checking whether last time's problems got fixed.
Do the math
People ask me if a quarterly drill is worth the hassle. I ran the numbers.
One drill: 12 people for half a day. Call it a little over two grand in labor. Hurts to watch.
But here's what a blind recall costs: every extra day product sits on hold in a cold building — rent, labor, customer claims — we estimate high four figures to five figures a day internally. And if you can't pinpoint the lot and have to widen the recall, pulling good product off the market — product value, freight, destruction — starts in the tens of thousands, before you count the trust you burned.
The expensive version is Blue Bell. In 2015, their Brenham, Texas plant tested positive for Listeria in February — the kind of situation the FDA treats as a Class I recall, reasonable probability of serious health consequences or death. The first recall notice didn't go out until March 13. They didn't recall all products until April. Ten people got sick. Three died. In 2020 the company pleaded guilty — $19.35 million in fines and settlements, which the Justice Department called the second-largest food-safety resolution in U.S. history. Slow and secretive cost more than the contamination itself.
After that math, nobody asks whether the drill is worth it.
A note to fellow operators
Honestly, a recall drill doesn't test speed. It tests your Tuesdays. That one extra scan at receiving, that one extra line on the putaway tag — that's what saves you when it matters.
Two reminders for folks in the same seat. First, the FDA's FSMA 204 traceability rule requires high-risk foods to be produced as a sortable electronic spreadsheet within 24 hours of a request — the compliance date got pushed to July 2028, but the requirement didn't change. Our two-hour target isn't for show. It's because when it really happens, the customer wants an answer in two hours. Twenty-four is too late.
Second, the hardest part of a drill isn't finding the bad lot. It's proving the other lots are clean. The FDA wants the full one-up-one-down chain. One broken link and your "clean" means nothing. Keeping the chain intact on ordinary days matters ten times more than running fast on drill day.
Folks, don't wait for the FDA to knock before you practice. By then it's too late.
By Chen Feng, Warehouse Operations Manager
