Fall 2023, a 140,000-square-foot food distribution center outside Kansas City, 2:47 a.m. The night-shift supervisor called me, and I could hear the shake in his voice. Eight stacker cranes sat frozen mid-aisle. The task queue on the screen hadn't moved in twenty minutes — the warehouse control system had just taken a patch, and its task scheduler was dead. The lights were on. The power was fine. The cranes weren't broken. The system's brain was just scrambled. Meanwhile the WMS kept pumping waves down, and outbound trucks were already lining up at the dock doors.
Let me be clear: this is not a power outage. A blackout is almost a relief — the generator kicks in, everyone knows something big happened, and the playbook is obvious. A software failure is worse because it's half-alive. Everything looks fine. Nothing moves. And the night crew's first instinct is always "let's wait and see if it fixes itself." I've watched a warehouse ride that instinct from 2 a.m. to 5:30 a.m., miss its first delivery appointment window, and spend the morning paying detention fees and rescheduling. Waiting is the most expensive option on the menu.
First, figure out which layer actually broke
An AS/RS runs on three layers: the WMS issues orders, the WCS schedules them, and the crane PLCs execute. Which layer broke dictates everything you do next, so don't just start rebooting things.
A frozen WCS is the most common case: the queue stalls while every crane shows online, sitting in place waiting for instructions that never come. A PLC-layer problem usually hits one crane with an alarm while the other aisles keep running. Network flapping is the nastiest — tasks drop in and out, and the floor looks haunted.
My most expensive lesson came in 2019 at a warehouse in Ontario. The WCS looked dead, and the shift manager rebooted the server without a second thought. Every in-flight task lost its state. Dozens of pallets no longer matched their system locations, and it took three full days of cycle counting to reconcile. Three days of manual outbound verification at a fraction of normal throughput. My iron rule since then: read the logs before you touch anything. A reboot is the last resort, never the first move. Before you act, confirm three things: the timestamp of the last successfully dispatched task in the WCS monitor, whether every crane PLC is still online, and whether the WMS-to-WCS interface heartbeat is healthy. Ten minutes of checking can be the difference between a thirty-minute fix and a three-day mess.
Drain the queue before you do anything else
Once you've confirmed it's a WCS-layer problem with healthy equipment, the first step isn't fixing — it's draining. Let the tasks already dispatched to the cranes finish naturally, while freezing the WMS from releasing anything new. The order matters: pallets on forks and conveyors belong to someone. A hard stop abandons them mid-journey, and good luck finding them later.
During the drain, the floor does exactly three things: freeze wave releases in the WMS, allow in-flight tasks to complete on their own, and manually log the position of any pallet stuck on a conveyor. It feels slow — usually tens of minutes — and someone will always pressure you to hurry. Don't. What the drain buys you is inventory accuracy. Remember the Ontario warehouse: they skipped a thirty-minute drain and paid for it with three days of reconciliation. That's a trade-off you only need to learn once.
How degraded mode actually runs: semi-auto and manual
When the WCS isn't coming back anytime soon but today's trucks can't wait, you run degraded. There are two gears, depending on how many people you have.
First gear is semi-automatic: switch the cranes to standalone mode and have an operator at each aisle mouth dispatch putaways and retrievals from printed task sheets. Throughput drops to roughly a third to a half of normal, but the flow never stops, and trucks keep loading one after another. Second gear is fully manual: forklifts work the aisle ends, operators pick by hand, and every move gets entered as a manual inventory transaction in the WMS. It sounds primitive, but that night in Kansas City, semi-auto carried us for six hours and we hit the day's outbound plan without missing a single truck.
Two details decide whether degraded mode saves you or buries you. First, the WMS has to switch into manual-task mode. If it keeps releasing waves on an automation cadence, tasks pile up faster than the floor can clear them — you're pouring fuel on the fire while trying to put it out. Second, log every manual move: who touched it, which pallet, from which location to which. When the WCS comes back up, you'll reconcile physical against system. Without that handwritten trail, the discrepancies stay buried until the next physical count — and by then nobody can tell you who moved what wrong.
The drill checklist: how often is often enough
A degraded-mode procedure you've never rehearsed doesn't exist. At 2 a.m. during a real failure, nobody is reading a forty-page manual. It's all muscle memory.
My recommendation: a tabletop walkthrough every quarter — shift manager, WCS vendor, and IT in one room, thirty minutes, talk through the whole sequence. And one live drill a year: pick a low-volume night shift and actually run manual mode for two hours. The live drill always surfaces something broken. I've never seen a first drill go smoothly.
Before any of that, get these ready:
- The WCS vendor's emergency contact printed and taped to the control room wall — not just saved in someone's phone, because at 2 a.m. that someone might not pick up;
- Blank manual task-sheet templates pre-printed and sitting at the floor station — don't go hunting for a printer mid-crisis;
- The WMS manual-mode switchover steps on a single sheet, taped next to the console;
- One named person — and only one — authorized to order the drain and the degraded switchover, because three people giving orders at 3 a.m. is its own disaster;
- After every drill, record one number: minutes from fault detection to degraded mode running. Write it down. That's your ammunition the next time you ask the boss for budget and headcount.
So here's the question for you: the last time your WCS failed, how long did it take to get degraded mode running? If the answer is "we've never tried," then the next time the phone rings at 2 a.m., the first thing to panic won't be the equipment. It'll be the people.
