Run a 50-AMR warehouse and your RCS can easily spit out thousands of alerts a day. Think of it as installing 20 apps on your phone, each pushing 10 notifications an hour — within a week, the control-room operator learns to dismiss them all on reflex. And that is when a real fault drowns in the noise, unseen by anyone.
This is alert fatigue. Hospitals figured it out a decade ago. Warehouse automation is next.
Why RCS Alerts Explode
The RCS is the fleet's traffic cop and dispatcher — routes, tasks, charging. Think of it this way: every AMR carries dozens of sensors — LiDAR, safety bumpers, encoders, the battery BMS, WiFi radios — each with its own threshold, each firing events upward. Fifty robots times dozens of sensors times 24 hours: that is your raw material.
The flood usually comes from three places.
First, the robot layer. Events the AMR itself throws: battery temperature wobbles, one-off obstacle slowdowns, relocalization, WiFi roams. High volume, and the vast majority self-heal.
Second, the fleet layer. Events the RCS throws: task timeouts, path deadlocks, zone congestion, charge-queue waits. Low volume, but every single one touches throughput directly.
Third — and I will say this bluntly — the integrator's cover-your-backside instinct. Plenty of projects ship with alert rules configured for liability rather than operations: "pop up when WiFi signal drops below -70 dBm." That alert does nothing except train operators to go numb. I have zero patience for pseudo-requirements: more alerts is not more diligence. A thousand alerts nobody reads is the same as zero.
Triage: AMR Alarms Get Three Levels. More Than That Is Theater
Three levels: P0, P1, P2. One hard line separates them: if nobody looks at this alert, does the line stop within 30 minutes? Yes goes to P0. No, but throughput takes a hit, goes to P1. Neither goes to P2, straight into the log. Crude, but it works.
| Level | Typical alarms | Delivery | Response |
|---|---|---|---|
| P0 — life or death | E-stop pressed, safety bumper hit, LiDAR blocked, robot offline, battery overtemperature, main-aisle deadlock | Siren plus SMS/phone call | Someone responds within minutes |
| P1 — throughput at risk | Failed return-to-charge, single nav failure (auto-recovered), charger queue, zone congestion slowdown | Control-room list plus shift summary, no popups | Handle within the shift |
| P2 — log only | WiFi roams, speed derates, heartbeat jitter, one-off obstacle slowdowns, self-healed events | Log only | Dig up when troubleshooting |
Two stories from the field.
Late 2024, ahead of the Singles' Day peak, an e-commerce warehouse in eastern China ran 60 AMRs with the RCS alert policy at factory default: everything on, more than 4,000 alerts a day. The operator later told me he had learned to "glance at the popup with peripheral vision and click it away." Then one robot's BMS flagged battery overtemperature — a genuine P0 — buried under thousands of "WiFi roam" and "obstacle slowdown" entries. More than forty minutes later a floor walk found the robot dead in the main aisle, half a picking line down. The postmortem fit in one sentence: it was not that there were no alerts; there were so many that it was the same as none.
The counter-example sits in the south: a 3C warehouse with 40 AMRs spent its first month pushing popups for "relocalization succeeded" — self-healed events. Seven hundred of its 800 daily alerts were good news. New rule: self-healed goes to the log; only failed recovery escalates. Popups dropped from 800 to about 60 a day, and the operator started watching the screen again. The problem was never "not enough alerts."
Muting Is Not Deleting: Three Pitfalls in Alert Tuning
Levels are set. Now comes the hands-on work of muting rules and tuning thresholds — and three pits I have watched swallow at least one project each.
Pit 1: mute it and forget it. Muting is a risk decision. It needs a change record: who approved it, why, and when it expires. Think of it as pulling the battery out of a smoke detector without telling building management — when there is a fire, the first question is "who pulled the battery." My rule: every mute goes into a register with an approver and a review date; expired mutes auto-restore, forcing a re-evaluation.
Pit 2: copying thresholds from someone else's warehouse. Floor flatness, WiFi coverage, rack density — no two sites match. "Alert when obstacle slowdowns exceed 10 per minute" might be perfect at Site A and a false-alarm machine at Site B. The right sequence: run the first two weeks with everything logging but nothing pushing, then set thresholds from your own data. Observe first, legislate second — not the other way around.
Pit 3: over-aggressive dedup. Some teams merge "navigation failure" and "task timeout" into a single "AMR anomaly" to cut the count, then cry during troubleshooting. Only dedupe repeats of the same cause, on the same robot, within minutes of each other. Never merge different causes — you dedupe noise, not information.
Before muting any single alert, run it through this decision matrix:
| Minor consequence | Severe consequence | |
|---|---|---|
| Frequent | Mute, log only | That is a design defect — fix the system, not the alerts |
| Rare | P2 log | Keep it, P0 push |
The top-right cell deserves a word: an alert that is both frequent and severe means your system itself is sick. Tuning alerts there is giving painkillers to a critical patient.
And one counterintuitive take to close the technical part: people assume "the more you mute, the more dangerous the system." Backwards. Muting is not dangerous; what is dangerous is muting things nobody knows are muted. A system with everything at P0 and a system with no alerts die exactly the same way — nobody is watching.
So was the money worth it? One number tells you: the average time from a P0 alert firing to a human starting to work it. Get that from 40 minutes down to 5, and your alert-governance effort paid for itself. If it does not move, everything before it — the triage, the muting, the dedup — was theater.
By Da Liu, Automation Engineer
