Bottom line first: in the first year after a WMS go-live, roughly six in ten production incidents are integration-related — and seven in ten of those are "it was down for hours and nobody knew." Monitoring and alerting is not something you bolt on after launch. It belongs on the go-live night checklist, delivered alongside the interfaces themselves. This piece walks through the train wrecks I've actually seen, then gives you a configuration checklist you can copy directly.

Why a dead interface is so dangerous

The WMS itself rarely crashes. Databases have backups, apps run in clusters, and when a server dies somebody screams. The terrifying scenario is both sides believing the other is fine: the ERP thinks shipping labels are printing, the TMS thinks status updates went through, while the WMS quietly fails a scheduled job — again, and again, and again. No alarm. Only a queue growing deeper.

Late 2021, I was on a 3PL go-live in Suzhou. We integrated a carrier's label-printing API. Go-live night went perfectly. At 3 AM on December 2, the carrier renewed a certificate, and our label API started throwing TLS handshake failures. The WMS retried three times, wrote failure logs, and moved on — in complete silence. At 8 AM, operators found no labels had printed since 3 AM. Over 900 parcels were piled up, drivers waiting in the cold for ninety minutes. That day cost the client nearly ¥20,000 in penalties and overtime. The fix took ten minutes — a certificate trust setting. Five hours of downtime, because nobody was told.

I've seen this exact hole three times. Another: 2023, Dongguan, an e-commerce warehouse. The ERP-WMS inventory sync died. Nightly reconciliation failed, and the warehouse oversold for four straight hours. The CS manager asked, "Is your system buggy?" I said, "No bug. Nobody was watching it."

So here's the counterintuitive take, up front: more monitoring is not better monitoring — the goal is that when something breaks, the system knows before the customer does. You don't need a dashboard with 200 metrics. You need the one alert that wakes you up at 3 AM. The whole configuration comes down to one sentence: everything that should ring, rings; nothing that shouldn't, does.

How to do it: four steps to a working setup

Step 1: List the interfaces that MUST be monitored

Don't monitor everything. Protect the vital organs first. These five categories are non-negotiable:

  1. Label/shipping-document APIs: when they die, the warehouse stops. This is P0, full stop.
  2. Outbound/inbound confirmation callbacks (ERP/TMS write-back): when they die, data drifts — overselling and lost orders start here.
  3. Inventory sync interfaces: scheduled reconciliation jobs. When they die, you're flying blind.
  4. Master-data sync (SKUs, locations, vendors): won't explode immediately, but rots slowly — new products never arrive, old ones never retire.
  5. Print/hardware middleware (printers, scales, scanner gateways): the easiest to forget because they're "not system interfaces," but when they die the floor stops instantly.

Add a line to the go-live night checklist: every one of these five categories must have a monitoring item and a named alert recipient. No blank rows allowed at sign-off.

Step 2: Set thresholds — better to wake you up than miss it

The classic rookie mistake is guessing thresholds. Here's a starting set. Deploy with these, then tune after two weeks of real data:

  • Consecutive failures: the same interface failing 3 times in a row triggers an alert. Why 3? One failure could be a network blip, two could be the other side restarting, three almost certainly means something is genuinely wrong.
  • Response time: for APIs that normally answer under 500 ms at p95, set the threshold at 2 seconds; for heavy ones that normally take 1–2 seconds, set 5 seconds. Thresholds should be roughly 3–4× normal behavior — never one fixed number for every interface.
  • Queue backlog: retry queue or pending messages above ~50 items for 10 minutes straight — alert. Backlog is the signature of slow death, and it's more worth watching than momentary failures.
  • Scheduled jobs: if a job hasn't started or finished 15 minutes past its scheduled window — alert. For reconciliation and sync jobs, "didn't run" is exactly as dangerous as "ran and failed."
  • Certificate/key expiry: alert at 30 days and again at 7 days before expiry. That 3 AM Suzhou incident? A cert renewal nobody knew about.

These are starting values, not standards. Every project's interfaces behave differently. Watch two weeks of monitoring data after go-live, then fine-tune. The target: false alarms under twice a week, missed real incidents at zero.

An overhead scanner scanning cartons on a conveyor

Step 3: Tier your alerts — if everything is urgent, nothing is

I've seen this hole three times: alerts without tiers are the same as no alerts. When every alert arrives by SMS, the on-call engineer starts ignoring them on day three, mutes the thread on day five — and then the real incident lands.

Three tiers, written in stone:

  • P0 Critical (phone call + SMS): label API down, consecutive confirmation-callback failures, queue backlog over threshold. Response: someone acknowledges within 15 minutes.
  • P1 Important (SMS + chat tool): scheduled job timeouts, response-time breaches, master-data sync failures. Response: handled within 2 hours.
  • P2 Notice (email / group message): cert expiring in 30 days, occasional non-critical failures. Handle on a workday.

One more hard rule: if a P0 alert goes unacknowledged for 30 minutes, it auto-escalates to the project manager. And build a real on-call rota — holidays included. "Everyone's in the group chat watching" is not a plan.

Step 4: Decide who gets the alerts — the recipient list matters more than the monitors

Monitors configured, alerts landing in a dead inbox — that's zero. The worst case I saw: alerts were going to the mailbox of an implementation consultant who had left six months earlier. An interface was down for two days and nobody in the whole project team got a thing.

Set the recipient list like this:

  1. Every interface gets a primary owner (usually the implementation consultant or ops engineer) plus a backup.
  2. At least two delivery channels: SMS + chat tool, plus phone for P0.
  3. When someone leaves or the rota changes, the list updates the same day — make it part of the handover checklist.
  4. Once a quarter, run an alert drill: deliberately kill a test interface and verify the whole chain — does the alert fire, does it reach the right person? An untested alert chain does not exist.

Common crash sites: run through these the night before go-live

  1. Monitoring "is it up" but not "is it right": the API returns 200, but the payload is empty or wrong. I once saw an inventory sync "succeeding" daily while pushing all zeros — a bad WHERE clause in the extract query. So critical interfaces need business validation: after a successful callback, spot-check one record; if the numbers don't reconcile, alert.
  2. Retry policies hiding the avalanche: retry 5 times, one minute apart. When the other side dies, the WMS silently retries, the logs fill with "retrying…" and nobody is alerted. Retry and alerting are different things: log on the first failure, alert on the third — don't wait for retries to exhaust.
  3. Monitoring production jobs but not the scheduler itself: the scheduler dies, nothing runs — and every job monitor shows "last successful" because nothing triggered. Add a heartbeat monitor: the scheduler writes a heartbeat every 5 minutes; no heartbeat for 10 minutes is an instant P0.
  4. Alert messages written like riddles: "ERROR CODE 5003." The on-call engineer climbs out of bed, reads that, and the first thing they do is call you. Every alert must include: which interface, what's wrong, what's impacted, and what to do first. Here's a template: "[P0] Label API failed 3× consecutively (last at 03:12), labeling operations impacted. Check carrier API status / certificate; failure logs at ."

Closing: go-live night checklist (integration monitoring edition)

Print it, stick it on the wall, check every box:

  • All five must-monitor interface categories listed, each with a monitoring item
  • Thresholds set at ~3–4× normal behavior; 3 consecutive failures triggers an alert
  • Alerts tiered P0/P1/P2; P0 calls the phone; unacknowledged P0 auto-escalates at 30 minutes
  • Recipient list current, at least two channels, backup named
  • Business-data validation added on critical interfaces — not just "is it up"
  • Scheduler heartbeat monitor configured
  • Cert/key expiry alerts at 30 days + 7 days
  • One full alert drill completed end-to-end

A dead interface isn't the scary part. The scary part is nobody knowing. This setup is an afternoon's work — and it blocks most year-one production incidents. Don't wait for the first train wreck. Its tuition would pay for ten setups.

By Lin Xiao, WMS Implementation Consultant