Live AI Operationsaihabits.ai

Live AI Operations

Demonstrated, Validated, Not tested

The three lines every stop test should end with, and why the last one matters most.

Last week I wrote that a revoke is not a stop. This week is about what you can honestly say after you’ve tested one.

When a board hears “we tested the kill switch,” it usually hears more than was tested. Someone pressed a button in staging, the agent went quiet, and the sentence travels upstairs as “we can stop it.” Nobody lied. The report just never said where the test ended.

So I’ve started ending every test report with three lines.

Demonstrated

What you actually saw happen, with a clock running.

Here’s mine. On Friday night, Oct 16, I ran a stop drill on a synthetic billing agent in my own sandbox. No real customers, no client system. I wanted to watch my own drill fail before asking anyone else to run one.

  • I pressed STOP. The main agent halted 5.9 seconds later. The runbook target was under 5.

  • 41 actions happened after the stop: 4 reversible, 7 fixable after the fact, 30 permanent.

  • The last one came 58.1 seconds after I pressed stop.

Most of those 41 didn’t come from the agent I stopped. A mail relay kept delivering what was already queued. A second agent kept working through tasks it had been handed. Five jobs the first agent had scheduled fired on time. The relay didn’t even know which agent had been stopped.

That’s the Demonstrated line. It’s specific, it’s timed, and it includes the part that didn’t go well.

Validated

What you checked without pressing anything live: walking the escalation path as a tabletop, matching it against the logs, confirming the names on the stop list are real people who can be reached.

In my drill, the stop list named two people: me as the primary and a backup stopper. I confirmed the list existed and I used it. That’s as far as it goes. Having a name on a list is validated. It isn’t demonstrated.

Not tested

What the drill didn’t touch, written in plain words.

From my own run:

  • The backup stopper never pressed STOP.

  • Nobody outside the stop list tried to stop it, so I don’t know if a stranger’s stop would be refused.

  • The agent wrote one invoice for $7,800 against a $5,000 approval limit. Zero alerts fired, because the sandbox had no runtime monitor. I found it reading the logs afterwards.

  • No live production stop. That’s true of every drill run on a mirrored copy, and it should say so.

Nobody enjoys writing this line. It’s the one a board most needs to read, because everything on it is something the company is currently assuming.

Stop & Undo Desk: try this on one agent

Pick one agent you run and write the three lines for it today, before any drill.

  1. Demonstrated: when did someone last press stop on it and time it? If the answer is never, this line is empty, and that’s the finding.

  2. Validated: who is allowed to stop it, and have you confirmed each of them can be reached at 2 a.m. on a Saturday?

  3. Not tested: list what the agent hands off, schedules, or sends through something else (a mail queue, another agent, a job scheduler). Each of those is a place a stop might not reach.

If line 3 is longer than line 1, you know where to start.

The point

My runbook claimed four things. The stop would be immediate. Nothing more would happen after it. Handed-off work would stop. Scheduled jobs would be cancelled. The drill showed a gap on all four.

I’d rather find that on a synthetic agent on a Friday night than hear it from a customer on a Monday morning. A report that leaves out the Not tested line is asking the board to assume.

Synthetic self-drill in my own sandbox, not a client system. Personal view, not legal advice.

Below the line for paid subscribers: a fill-in template for the three lines, my full sandbox run written into it, and the one line that turns a finding into a decision someone owns.

This is the public preview of a longer issue. Read the rest on Substack.