The Stop Button Worked the Whole Time. Nobody Pressed It.

There was a clear stop mechanism. On paper.

A mid-sized company had introduced an AI system for quote calculation. The managing director had announced the project at the company assembly as a great step forward; there was even a small press release. And there was, properly documented, a person with the authority to stop the system if something went wrong.

After a few weeks, a clerk noticed that the calculations for certain special machines were systematically too low. Not dramatically (a few percent), but reliably. She told her supervisor. He told nobody. Not out of malice, but because he understood exactly what it means to slow down a project the boss has just celebrated in front of the entire workforce.

The system ran for almost a year and produced a series of quotes that cost the company real money. The stop button worked the whole time. Nobody pressed it.

Governance is decided before the first AI project

This is the most underrated part of AI governance. We write down who is allowed to stop, and we mistake that for control. Whether anyone actually stops is not decided by the documented right; it is decided by one question: does this organisation treat the person who speaks an uncomfortable truth as a troublemaker, or as someone doing their job? And that answer is fixed long before the first AI project starts.

The honest test

Think of the last genuinely bad news that made it to the top of your company. Who delivered it, and what happened to that person afterwards? Were they heard and backed, or were they quietly labelled “difficult”? That one memory says more about your AI governance than any policy document. An organisation that punishes the messenger today will not stop an AI tomorrow, however cleanly the stop right is documented.

• Make drift visible by default. Deviation dashboards that the team sees, not just the project sponsor.
• Reward the first report. The clerk who spots the anomaly saved you money. Say so, publicly.
• Rehearse the stop. A kill switch that has never been used in an exercise will not be used in an emergency.

Want to know whether your organisation would press the button in time? Book a demo or try our ROI calculator.