Four ways an AI system fails
without telling you.
A system that crashes tells you. Almost nothing fails that way. These four keep running, keep producing output, and stopped being true weeks ago.
A system that crashes tells you it crashed. That's the easy case, and almost nothing you install will fail that way.
The expensive failures keep running. The job fires, the report arrives, the dashboard loads, and the thing inside it stopped being true weeks ago. Nobody notices because nothing broke.
Four failure modes, all of them survivable, none of them detectable by watching whether the system is up. Each one has a check that costs minutes.
The four modes
It's scheduled, and it's failing every single run
The most common one and the least visible. A scheduled job can be registered, enabled, and listed as active while failing on every execution. The system that runs it considers "I tried" to be success.
We lost roughly three weeks of a daily report this way. It was loaded. It was scheduled. It hadn't delivered anything in twenty days and nothing anywhere said so.
The check: look at the exit status of the last run, not whether the job is enabled. Those are different fields and only one of them means healthy. Then confirm the output landed where it was supposed to land.
It's succeeding on input that went stale
The job runs, the connection holds, the output generates. Upstream, a permission narrowed or a field got renamed, so one source now returns nothing. The system treats empty as a valid answer and reports confidently on what is left.
The check: pick one number in the output and trace it to the source record by hand. Once a month. If a section has quietly gone empty, this is the only thing that finds it.
The quality drops without the behaviour changing
Output still appears, on time, in the right format. It's just worse. There's no error state for worse.
This is the failure with no bounce-back. Whoever receives the output is the only one who sees it, and they usually respond by quietly valuing your work slightly lower rather than telling you.
The check: grade a fixed sample on a schedule against a written standard. Not when someone complains. The six-point check is the version of this we run.
Something hand-written is quietly overriding the live data
The subtle one. Most systems mix live inputs with something a human set once: a priority, a note, a flag, a default. Over time the human line stops being true, and because it was written deliberately it usually wins.
Ours once told us to chase a deal that had closed six days earlier. Both facts were in the system. The stale hand-written line outranked the live one.
The check: list every value in the system that a person set by hand and give each one a date. Anything older than the thing it competes with is now a liability. Never let a field that updates on its own be overridden by one that doesn't.
Why "is it working?" is the wrong question
All four modes pass a liveness test. The process is running, the schedule is registered, the page loads, the file has today's date on it.
Every one of them fails a truth test. Did the output arrive where it was meant to, does one traced number still match its source, has quality been graded lately, and is the human-written part younger than the live part.
Four questions, once a month, roughly twenty minutes. That's the entire discipline, and it's worth more than any monitoring you can buy.
What to demand at handover
If someone is building this for you, these four belong in the scope before you sign. They're cheap while it's being built and expensive to retrofit.
- A failure that's loud. When a run fails, something reaches a human. Silence must mean success, not absence.
- A visible last-success timestamp. Not last-attempted. On the thing you already look at, so a stale date is obvious without going hunting.
- An empty-source alarm. If a connected system returns nothing, the output says so out loud rather than rendering a confident partial answer.
- A dated list of every hand-set value. One page. It's the cheapest document in the build and the one that ages worst without it.
Where to start
Check the exit status today
Enabled isn't healthy. Find the last run's result and confirm the output actually arrived somewhere you can see it.
Trace one number by hand
Pick a single figure and follow it back to the record it came from. Ten minutes, and it's the fastest way to find a section that went quietly empty.
Put the four in the scope
Loud failures, a last-success timestamp, an empty-source alarm, and the dated list. Cheap now, expensive later.
Why we know these four
We've hit all of them on our own systems, which is the only honest reason to publish a list like this. The three-week silent job and the stale line that outranked live data were both ours, and both are now checks that run by default on everything we install.
MarqueeOS builds the system and the thing that tells you when it stops being true. The four checks above work whether or not we built it.
Keep reading
The 6-Point Check For AI Work Before It Leaves Your Company
Most AI failures don't look like errors. Here's how to grade output you didn't build, and turn it into an error rate you can actually act on.
Read the guide → Operations · 6 min readThe 30-Minute AI Inventory: What Is Running In Your Business And Who Owns It
Four questions, four groups, one specific order. Surfaces the undocumented setup carrying real volume before the person running it resigns.
Read the guide →Something running that
nobody has checked?
Thirty minutes. Bring one automated thing you rely on and we'll run the four checks against it live.
Book a 30-minute call →