Bottom line up front: for any WMS or automation go-live, set up four alert tiers — Tier 1 (SMS + phone call), Tier 2 (@mention in the WeCom on-call group), Tier 3 (plain WeCom message), Tier 4 (email digest) — with response SLAs of 5 minutes, 30 minutes, 4 hours, and next-day. I have seen the "send everything by SMS" approach three times. All three died of alert fatigue: the on-call person stopped reading at message 47, and message 48 happened to be the real fire.
Alert tiering is an implementation problem, not an ops problem
Most projects treat alerting as "something ops figures out after go-live," and the implementation consultant walks away. After 30+ go-lives, I will say it plainly: alert rules must be locked down during UAT, because tiering defines your incident response process, and incident response is part of the WMS blueprint that gets signed off.
November 2023, a cross-border e-commerce client in Shenzhen, second month after go-live. At 2:14 AM, an inventory freeze alert fired because the integration queue was backing up — and it went out by email only. The on-duty supervisor had Do Not Disturb on; the email sat unread in his inbox. By 8 AM, over 2,000 orders couldn't ship because inventory was frozen. Customer service got crushed. Direct damage: roughly 18,000 unshipped orders that day, at an average order value of $22 — close to $400,000 in daily revenue gone, before marketplace late-shipment penalties. I have seen this exact failure three times. Every time it was "the alert fired, but nobody looked."
Here is the counterintuitive part: missed alerts are a tiering problem 90% of the time, not a channel problem. I have also seen the opposite extreme — a 3PL in the eastern US whose cold-chain temperature probes sent an SMS every time the reading wobbled, 60+ texts a night. By day three the on-call manager blacklisted the alert number. Too many alerts equals no alerts. Tiering exists to protect the on-call person's attention so that only genuinely life-threatening events wake them up.
Four steps to build the escalation chain
Step 1: Define the tiers first — no more than five Tier 1 categories.
Tier by two dimensions only: blast radius × time-to-irreversible-damage. Tier 1 means "if nobody acts now, there will be a production incident within the hour." My standard client template:
| Tier | Definition | Channel | Response SLA | Examples |
|---|---|---|---|---|
| Tier 1 | Production incident within 1 hour if ignored | SMS + phone call | 5 min | WMS down, cold-chain power loss, fire alarm integration |
| Tier 2 | Partial impact, can hold for hours | WeCom @on-call group | 30 min | Interface backlog over threshold, mass AGV offline |
| Tier 3 | Needs action, not urgent | WeCom plain message | 4 hours | Label printer out of paper, single PDA failure |
| Tier 4 | Stats and trends | Email digest | Next day | Yesterday's incident summary, disk at 70% |
Keep the Tier 1 list short. My iron rule: no more than five categories. More than five means you couldn't tell priorities apart, which means you didn't tier at all.
Step 2: Build the on-call rota — nobody on duty more than two weeks straight.
Use a primary + backup pair: the primary responds, the backup only gets a phone call if the primary doesn't acknowledge within 10 minutes. Rotate weekly; two weeks max. I once saw a client keep one person on call for a month straight — by week three his response time had clearly degraded. Asked why, he said he'd gotten into the habit of assuming every alert was a false positive.
The rota must live in the system: the alerting platform (WeCom bot, SMS gateway) routes to whoever is on duty by date automatically. No manual forwarding. Manual forwarding is the biggest single point of failure — in March 2024, a 3PL in eastern China had its on-call supervisor on leave; the SMS went to his phone, he was on a high-speed train with no signal, and the backup didn't even know he was the backup. That was a Tier 1: an AS/RS stacker crane fault, line down for 3 hours, 9,000 cases short-shipped.
Step 3: Build the escalation chain — auto-escalate on timeout.
This is the part most often skipped and most likely to save you. The rules are simple:
- Tier 1 fires; if the primary hasn't hit "acknowledge" in the system within 5 minutes, auto-call the backup;
- 10 more minutes with no acknowledgment, auto-call the operations manager;
- 15 more minutes, call the general manager directly and @everyone in the WeCom management group.
Log every escalation: who acknowledged at what time, what action they took. The log isn't for blame — it's for the post-mortem. An alert process with no records will fail the same way next time.
Step 4: The go-live-eve checklist — test alerting like a feature.
This is my mantra, and it's what I do the night before every go-live:
- For each Tier 1 category, trigger it manually and confirm SMS + call reach the on-duty phone within 1 minute
- Simulate the primary not picking up; confirm auto-escalation to backup at 5 minutes
- Confirm the WeCom bot token is the production one (test bots post to test groups — switch it at go-live)
- Confirm the SMS gateway has credit (laugh if you want — I've had a client whose texts stopped on day three because the account ran dry)
- Confirm the rota covers the next 4 weeks and routes correctly in the system
- Confirm the Tier 4 digest recipient list includes the client's IT lead
Three failure modes I've seen more than three times
Failure mode 1: Test and production alerts mixed together. Testers load-test at midnight, a flood of SMS alerts goes out, the on-call person gets woken three times and starts treating every alert as noise. Fix: test-environment alerts go to the WeCom test group only. Never touch the SMS channel.
Failure mode 2: The Tier 1 list keeps growing. Six months after go-live, every department lobbies: "ours is important too, make it Tier 1." Half a year later there are 20+ Tier 1 categories and the SMS channel is jammed again. Fix: tier-list changes go through review. Before I roll off a project, I write the review process into the ops handbook — adding a Tier 1 alert requires dual sign-off from the ops manager and the IT lead.
Failure mode 3: Fire and forget. The alert shows "sent" in the system, but the SMS gateway silently dropped 30%. Fix: run a monthly alert health check — messages sent vs. delivered. Delivery rate below 95%? Switch gateways. Put this check on the monthly ops meeting agenda. Don't rely on willpower.
Do the math: this whole setup costs a small warehouse roughly RMB 5,000–8,000 a year in SMS and platform fees, a large one RMB 20,000–30,000. One missed Tier 1 alert at 2 AM costs what the two cases above showed: $400,000 in a day's revenue gone, or a 3-hour line stoppage with 9,000 short-shipped cases. You can do that arithmetic yourself.
One actionable takeaway: if you're in WMS selection or implementation right now, put the "alert tiering plan" in the contract appendix and sign it with the blueprint. Don't wait until a 3 AM phone call to realize nobody ever decided this.
By Lin Xiao, WMS Implementation Consultant
