Think of it this way: on a conveyor, a carton passes the scan gate, the scanner does not beep, the operator scans it again — and the system now believes there are two orders. That is the problem API idempotency solves. The platform's network hiccups, a webhook times out with no acknowledgement, the platform dutifully retries three times — and without deduplication on the warehouse side, two phantom orders materialize out of thin air. The orders are fake; the picking, shipping, and returns are painfully real.
The principle first. Idempotency means: sending the same request once or ten times has the same effect. In engineering terms it comes down to one thing — the idempotency key. The platform includes a key with each order push; the warehouse side checks whether that key was already processed. Processed? Return the previous result verbatim and create nothing. New? Create the order and record the key. Think of it as the label on the carton: the scan gate reads the label number, not "how many times you scanned."
Idempotency Keys: Three Design Rules
Key design decides whether the whole thing actually works or is theater. Three iron rules. First, the key must carry business meaning. The best key is "platform + platform order number" — shopify_81204473 maps straight to a real order. The worst key is a random UUID: every retry mints a fresh key, which is the same as doing nothing. The most absurd case I have seen: someone used a timestamp as the key — retries a second apart produced different keys, and the dedup protected absolutely nothing. Second, key granularity must be right. Too coarse (store + date) and a legitimate second order gets killed; too fine (millisecond precision) and the retry never matches. Third, keys need a TTL. Never store keys forever — months later a customer places a genuinely identical-looking order (same platform, reused order number) and it gets wrongly flagged as a duplicate. Set the TTL to cover the platform's retry window; 24 to 72 hours is usually plenty. No platform retries months later.
Where Warehouse Dedup Lives: Three Tiers
Now the selection line: which layer does dedup live on? Three tiers. Tier one: a database unique constraint — a unique index on (platform, platform_order_no) in the orders table. This is the hardest wall; even concurrent double-writes bounce off it, and it costs nearly nothing. Its limit: it only guards order creation. Tier two: cache key plus state machine — store idempotency keys in Redis or similar and check before creating. The bonus: it also blocks duplicate state transitions (the same order marked "shipped" twice). The cost: one more component to babysit, and you must decide upfront what happens when the cache is down. Tier three: the platform sends the key itself — the cleanest option, but it requires upstream cooperation. Big platforms generally support it; small ones, it depends on the relationship. The dividing line is clear: modest volume, monolithic app — tier one is enough, do not over-engineer. Multiple systems, multiple platforms, serious volume — go straight to tier two. If you have leverage over the upstream, negotiate tier three.
A word on the tier-two contingency plan, because you will need it at 2 AM: when the cache goes down, you have two options — degrade to the database unique constraint (slower but alive), or refuse order creation outright (better to miss one order than to ship one twice). Which one depends on your business: an own-brand warehouse picks the former; a 3PL shipping on behalf of clients picks the latter — the compensation for one duplicate shipment costs far more than one missed order. Write this plan before launch, not while the cache is on fire.
A composite-key example: one 3PL integrated both Shopify and Etsy, and Etsy's order numbers can repeat across different shops — so the key had to be "platform_shop_order" in three segments. Miss any one dimension that can collide, and the whole thing is theater. Before launch, list every dimension whose combination must be globally unique, and verify them one by one. Do not wing it.
Case time. During a 2024 peak-season sale, a platform's webhooks timed out under load and retried three times within seconds. The warehouse OMS had no unique constraint, so three orders with the same platform number were created — and the night shift shipped all three. Two were recovered; the third was refused and returned. Return freight plus product loss on that one order would have funded the dedup feature for years. The other case is more infuriating: a 3PL integrated with multiple platforms used "platform order number" as the key — except one platform's order number changes between placement and payment (the temporary number is not the final number). Duplicates that should have been blocked sailed through; legitimate orders got blocked instead. Lesson: confirm the key's semantics with the platform, word by word. Never assume.
Three Maintenance Pitfalls
Three maintenance pitfalls. Pitfall one: TTL too short. A platform's delayed retry — say, 48 hours later — arrives after the key expired, and the duplicate walks right in. Pitfall two: you deduped creation but not state transitions. No duplicate order — but the "ship" endpoint got called twice, two shipping labels printed, the courier came twice. The money leaves anyway. Pitfall three: concurrent double-writes. Two retries land in the same millisecond, and the check-then-create sequence runs twice — both threads believe they are first. The fix: check-and-set must be atomic (SET NX and friends). Never hand-roll check-then-write.
One thing to get off my chest: people ask whether they need "end-to-end idempotency" on every endpoint. My answer: that is a pseudo-requirement. Read endpoints (GET) are naturally idempotent — adding keys there is pointless. Only write endpoints that change the world — create order, charge, ship — deserve the treatment. Spend the effort where it cuts.
Closing the books. The cost of one duplicate order: the shipped goods, reverse-logistics freight, customer-service hours — tens of dollars at the low end, hundreds at the high end. The cost of building idempotency: half a day for a unique constraint, one to two days for the cache approach. Whether the money is worth it comes down to one number: last year's duplicate count times average loss per duplicate. If that exceeds two days of dev cost, build it. I have yet to see the math say otherwise.
By Da Liu, Automation Engineer
