My verdict up front: if you are doing fewer than 3,000 orders a day, don't buy AI order screening. A rules engine will do the job.
That probably annoys a few AI vendors, but it is the conclusion I reached after watching several real deployments. The pitch is seductive: orders flow in, the AI flags the weird ones — stolen cards, suspicious addresses, bulk orders from a single buyer, high-return-risk addresses — auto-blocking or tagging them while humans only review the questionable ones. Sounds great. The problem is the accuracy math. The number on the vendor's slide and the number in your warehouse are often two different animals.
What the thing actually does
Let me define terms first. AI order screening does three things: it spots risk (does this order smell like fraud), it scores and ranks (highest risk first), and it assists the decision (block it, let it through, or send it to a human).
How is that different from a rules engine? A rules engine is "block any order over $500 shipping to a freight forwarder" — human-written rules, rigid but explainable. An AI learns patterns from your order history and finds combinations nobody wrote down, like "ordered at 3 a.m. plus new account plus high-value item plus billing address in another state."
But there is a catch that kills most small operations: the AI needs to be fed. Without tens of thousands of labeled historical orders — which ones were fraud, which ones were false alarms — the model is guessing. That is the core reason low-volume warehouses shouldn't bother. You simply don't have the data to raise a decent model. When a salesperson tells you "our pre-trained model works out of the box," ask one question: whose data was it trained on? Your categories, your ticket sizes, your customer mix — are they the same? Probably not. Discount the promised performance accordingly.
The number that matters is not accuracy
Vendors love to talk about "98% block accuracy." The two numbers you should actually care about are the miss rate and the false-positive rate.
A miss costs you directly: the stolen-card order ships, the money is gone, and you eat the chargeback fee on top. A false positive costs you indirectly, and it can cost more. I watched a DTC brand's warehouse go live with AI screening, and in the first month the false-positive rate was 4%. Sounds small, right? That was 4% of real orders getting stuck. Two full-time customer service people spent the month on reviews and apologies, and the labor cost came out higher than the old rules-plus-spot-check setup. The nastier cost is churn: some of the wrongly blocked customers just bought from somebody else. You can't put that on a spreadsheet, but it's real.
My rule of thumb: if your false-positive rate is above 2%, the system is losing you money. Don't buy the "the model keeps getting smarter" comfort line. It does get smarter — if someone keeps feeding it labeled data and tuning thresholds. Small teams don't have that person. The model's day-one performance is basically its ceiling.
Here is something most people miss: the biggest value of AI screening may not be blocking at all — it's triage efficiency. The good setup is "AI scores, humans review the suspicious ones," not "AI auto-blocks." Auto-blocking sounds like it saves labor, but you own every false positive. Let the AI do the first pass and let a person make the final call, so the chain of responsibility stays clean. Auto-blocking only makes sense in one narrow case: fraud signals so obvious and false-positive costs so low that you'd never be wrong — like fifty orders in an hour from the same device. Block those with your eyes closed.
Run it in shadow mode first
Whatever your volume, don't let the first thing the model touches be real orders. Run it in shadow mode: it scores, it logs "I would have blocked this," but it blocks nothing. After 30 days, reconcile its calls against human review — of the orders it wanted to block, how many were actually fraud; of the ones it cleared, how many slipped through.
Those 30 days are your negotiating leverage. The deck says 1% false positives; shadow mode says 5%? Now you know what the money buys. I watched one deal die right there — the vendor promised 99% accuracy, shadow mode showed 7% false positives, and the warehouse walked away before spending a dollar on implementation.
Shadow mode also buys your team time to learn the tool. Reviewers need to understand what the scores mean. Otherwise, on day one, "human review" becomes rubber-stamping whatever the AI says, and your review process exists only on paper.
Where I draw the line: 3,000 a day
Do the arithmetic. At 3,000 orders a day you're running over a million orders a year. Against publicly reported e-commerce fraud rates, that's tens of thousands of potentially risky orders a year. Human review can't cover that volume, a rules engine false-positives too much, and that's when AI starts paying for itself.
Below 3,000 a day? The volume is small enough that people plus rules cover it. My recommendation: get good at the rules engine first. A blacklist of bad addresses, amount thresholds, velocity rules, and the address verification your payment gateway already provides — those four moves catch the vast majority of common fraud. Spot-check the high-risk orders manually. When your volume grows and your data piles up, then talk about AI.
There's also a hidden line item in the ROI: implementation. AI screening isn't a SaaS login you turn on. It means wiring into your order system, wiring into your payment gateway, labeling historical data, tuning thresholds — two to three months of work is normal. For a small team, that effort is better spent getting the customer service process straight first.
One last practical thought. If you're under 3,000 orders a day and a salesperson tells you AI is the only thing standing between you and fraud, ask them this: how much are we actually losing to fraud per month, right now? Get that number. Then compare it against the annual AI license plus implementation cost. The answer will write itself. Don't buy the concept. Buy the math.
