The phone rang at 2 a.m. A shift supervisor at an e-commerce DC in Riverside was practically shouting: "Lao Mi, the sorter has lost its mind! The same carton just got scanned three times, the chutes are overflowing, the conveyor is about to jam!"
I told him to keep his hands off the equipment and pull two log files instead. Half an hour later he called back, sounding sheepish: "You were right... the WES released the same wave twice."
I've seen this movie too many times. The floor always says "the system is glitching." In my experience, nine times out of ten, what looks like a glitch is the WES and the WCS fighting each other — and the root cause is almost always the interface between them. Here's the troubleshooting chain I actually use, from symptom to root cause.
Don't Touch the Line. Get the Two Logs Talking First.
Command conflicts usually die in one of two ways: duplicate task dispatch, or a lost status callback. The WES sends a release command, the network hiccups, the acknowledgment never comes back, so the WES fires the same command again after its configured timeout. The WCS happily accepts both — because nobody built task-number deduplication into the interface. One task enters the sortation queue twice, and the same parcel gets requested by the chute two or three times.
Step one is always the same: pull the WES dispatch log and the WCS command-receipt log side by side, and line them up task number by task number. But there's a trap here that gets eight warehouses out of ten: the two servers' clocks aren't synced. The WES log says the command went out at 02:14:03; the WCS log says it was received at 02:13:47. When timestamps don't agree, you can never establish cause and effect. Before touching any log, confirm both sides are on NTP — and confirm the time zone. I've seen more than one site where one side logged in UTC and the other in local time.
I tripped over this myself back in 2021 at a DC in Ontario, somewhere north of a hundred thousand square feet, running a cross-belt sorter. Peak season, middle of the night: the same parcel showed up in the system three times. With both logs on the table, it was plain as day — the WES had released the wave twice, spaced exactly one timeout interval apart, and the WCS execution log showed two identical task numbers, both honored. Root cause in one sentence: timeout-based redelivery hitting an interface with no idempotency.
There's a genuine trade-off hiding in that timeout value. Set it too short and every network jitter triggers a false redelivery; set it too long and wave changes and rush orders sit around waiting. That number can't be a guess — it has to come from measured latency on your actual network, and the retry count and backoff policy belong in the interface spec in writing. Too many integrators leave this as a verbal agreement, and the agreement walks out the door with the engineer.
When a Rush Order Bumps a Live Task, Where Does the Old Task Go?
Rush-order preemption is the nastiest conflict zone, and the logic is trickier than plain duplication. The WES gets a rush order, marks part of the current wave's tasks as canceled, and slides the new tasks in. But the WCS already pushed those commands down to the PLC — the cancel arrives too late to stop a divert that's already in motion. Now the same parcel is being called to chute A by the old task and chute B by the new one. Which one does the equipment obey?
Troubleshooting means following two threads: the WES task state machine (did the cancel actually go out, and exactly when?) and the WCS command queue (where was that command in execution when the cancel landed?). The decisive evidence is still the timestamp delta — when the cancel was issued versus when the command actually finished executing. Do the arithmetic and you know whether the cancel had any chance of catching up.
There's a subtler trap underneath: the two sides don't even speak the same task numbers. The WES uses its own task IDs; the WCS translates them into equipment command numbers, with a mapping table in the middle. Lose that table and the two systems talk past each other. I saw it in 2023 at a warehouse outside Chicago — the mapping table lived in memory, the WCS service restarted overnight, and it was gone. The night-shift engineer stared at two logs and couldn't find a single matching task. Not because nothing was logged, but because the numbers didn't line up. The fix is unglamorous but effective: persist the mapping table to disk, and make the WCS log carry the original WES task number all the way through.
And here's my verdict on rush orders: don't do hard preemption unless your volume demands it. Give rush orders their own chute or their own wave — don't touch tasks that are already executing. The few minutes you save by preempting won't cover the cost of one line jam. If you must preempt, the cancel has to be two-phase: a pre-cancel that locks unexecuted commands at the WCS, then a formal cancel; anything already in motion goes through a compensate-after-completion flow. Expecting a single cancel command to outrun physics is just disrespecting the speed of light.
Stop Asking Why. Make It Idempotent.
Both failure modes above share one root cause: the interface isn't idempotent. What's required on the WES side is simple — every task carries a globally unique task number, and redeliveries keep the same number. On the WCS side: check "have I seen this task number?" on receipt; if yes, reply success and don't execute again. Status callbacks work the same way — attach a monotonically increasing sequence number so the WES can sort out-of-order or duplicate callbacks on its own.
At acceptance, I hand the integrator three sentences, and they go into the contract as an appendix: first, drop the network for a few seconds and reconnect — no task may execute twice. Second, restart the WCS — historical task mappings must survive. Third, canceling a rush order against an executing task must trigger a defined compensation action, not just "the cancel was sent." Pass those three and ninety percent of execution-layer ghost stories never happen.
My verdict stands: nine out of ten "system glitches" are interfaces without idempotency. Of the remaining tenth, half is unsynced clocks and half is somebody's shaky hand on a button at 2 a.m.
Before your next go-live, pull the network cable for a few seconds and plug it back in yourself. Only a system that survives those few seconds deserves your peak season.
