A week before last year's peak season, I was walking a 3PL in Rialto when their IT director proudly showed me the new carrier label API they'd just integrated. "Handles 2,000 requests per second," he said. I asked him one question, and he froze: "If this API goes down at 2 AM, who's the first person to know?"
He thought for a long moment. "Probably... the label printer operator at 6 AM?"
That tells you everything. How robust an integration is — that's a technical question. Who finds out first when it breaks — that's a management question. Most warehouses effectively have no integration monitoring. Not because they never set it up, but because what they set up might as well not exist: thresholds set absurdly, alerts routed to an inbox nobody reads, or 3 AM texts that wake the whole team until everyone mutes the channel.
In twenty years I've watched integrations die at least twenty different ways. Here's my monitoring and alerting checklist, laid out plainly. Follow it, and the first person to know about an outage will be someone on your team — not your customer.
The five signals. Everything else is a bonus.
Don't start with a fancy observability dashboard. In a warehouse, five signals cover you. Miss one, and peak season can still bite you.
First, heartbeat. Is the endpoint alive? Simplest and most skipped: ping the health-check endpoint every five minutes, or send a probe message. Carrier label APIs, address validation, duty calculators — anything third-party needs its own heartbeat. In 2023, a 3PL in Ontario lost four hours of label printing because a carrier's TLS certificate expired. No heartbeat probe, no warning. Nine thousand orders piled up before anyone noticed. Heartbeat checks cost essentially nothing. Skipping them is pure laziness.
Second, error rate. Failed requests as a share of total in a five-minute window. Share, not absolute count — during peak, volume multiplies tenfold, so absolute failures will rise normally. A rising share is what means real trouble.
Third, latency. P99 latency, not average. An average of 200ms sounds lovely until P99 hits eight seconds — meaning one percent of requests stall, usually clustered in the busiest minutes, and that's exactly what jams the packing line.
Fourth, backlog depth. Queue length, unprocessed file counts, retry-pool depth. "Slow but not dead" is the cruelest failure mode: everything looks fine on the surface while the queue grows deeper, and by the time you notice, 30,000 messages are buried and you can't dig out.
Fifth, reconciliation gaps. Scheduled reconciliation: orders we sent versus orders they received. Carrier labels, EDI 850/856 documents, inventory syncs — reconcile at least daily. Reconciliation is the last line of defense. When the first four signals all miss, this one catches it.
How to set thresholds: three tiers, no one-size-fits-all
The classic mistake is one threshold for everything: either the boy who cried wolf every day, or a real outage nobody notices. I tier by business impact:
P1 — phone call, respond within five minutes. Triggers: heartbeat fails twice in a row on a core interface (label printing, ship confirmation, inventory sync); error rate above 5% in a five-minute window; P99 latency above ten seconds for ten straight minutes. P1 means an actual phone call to the on-call engineer. Not a text, not an email. No answer in five minutes, it auto-escalates to the second contact.
P2 — text plus group message, respond within thirty minutes. Triggers: non-core interface (address validation, tracking callbacks) error rate above 3%; queue backlog above 5,000 messages and still climbing after fifteen minutes; more than fifty failed retries on the same batch. P2 goes to the on-call channel, and a "got it" reply counts as acknowledged.
P3 — email plus daily digest, handle during business hours. Triggers: reconciliation gap under 0.5%; a slow P99 creep that hasn't crossed the line; a certificate expiring in thirty days. These go in the morning report for the IT standup.
One critical detail: tune thresholds per interface, and freeze threshold changes two weeks before peak. The worst setup I ever saw had every interface sharing a 1% error-rate threshold. During peak, the address-validation API sat at 2–3% errors all day because shoppers type garbage addresses — alerts fired constantly, everyone ignored them, and that evening a genuine P1 outage got ignored as just another false alarm. Calibrate each threshold against that interface's historical P95 before peak. It takes an afternoon. Skipping it costs you the one alert that matters.
Who gets the alert: the on-call roster matters more than the tech
The tech can be perfect and still useless if alerts go to people who've muted everything. Three rules I insist on:
One, there must be an on-call roster — one named person, not "the team." An alert in a group chat means everyone assumes someone else will handle it. P1 alerts go point-to-point: first the on-call engineer, no response in five minutes auto-escalates to the IT manager, then to the operations director, who has authority to make business calls like switching to a backup carrier. The roster rotates weekly, published a week ahead, and holiday shifts earn comp time. A roster without comp time won't survive a month.
Two, separate channels by severity. P1 by phone, P2 by text and group chat, P3 by email. Never dump the P3 digest into the same channel as P1 — once the noise is high enough, people learn to ignore everything. I once saw a 200-person channel pushing 300 alerts a day. A P1 buried in there had zero chance.
Three, review the false alarms monthly. Pull last month's alerts: which were real incidents, which were false positives, which fired and got ignored. Any threshold with a false-positive rate above 30% gets retuned. Alerts ignored get investigated. An alerting system, like a person, rots without review.
And one lesson paid for in blood: run a "kill the network" drill before peak. One quiet weekend morning, cut the carrier API in staging and time everything — how fast the alert fires, how fast the on-call responds, how fast the fallback plan kicks in. A plan that's only been discussed on paper will fail its first real test, guaranteed. The warehouses I've worked with that drill versus those that don't perform like two different species on peak night.
What does this cost? Let me do the math.
Setting this up isn't expensive. Heartbeat probes and error-rate tracking with an off-the-shelf APM (Datadog, New Relic, or open-source Prometheus plus Alertmanager) takes about a week to stand up. The real cost is the on-call stipend: around here, a weeknight shift runs $50–80, weekends $100–150. Across a rotation, that's a couple thousand dollars a month.
Now the cost of not doing it. The most common scenario: the label API dies overnight, nobody knows until 6 AM, 8,000 orders backlogged. Every order delayed a day — missed carrier cutoffs, support tickets, marketplace penalties (Amazon OTDR drops half a point and your Buy Box weight takes the hit), plus overtime to catch up. One incident like that runs $20,000–50,000, conservatively. A single incident in a year pays for the monitoring ten times over.
So my verdict is blunt: integration monitoring isn't IT's elective. It's warehouse operations' required course. You can skip the fancy dashboards and the observability buzzwords, but five signals, three threshold tiers, and one on-call roster — those three should exist by tonight.
First thing tomorrow morning: ask your IT lead who finds out first when an interface dies at 2 AM. If they can't answer, you know what to do.
