How do I tell whether an AI pilot is worth keeping?

Run four tests. Switch it off for two weeks and see whether anyone complains. Compare its cost against the cost of the thing it replaced, not against zero. Trace every benefit number back to a measurement rather than a model. And ask whether it changed what your team can do, or only what that work costs. A pilot that survives all four is worth keeping. Most do not survive the first.

Why do most AI pilots fail to scale?

Usually because nobody decided who owns the output. The technology works in the demo, then stalls because no one is accountable for what the agent produces, no one checks it, and the benefit number was a model that nobody re-tested against reality. The blocker is almost never model capability.

How to tell whether an AI pilot is worth keeping

Switch it off for two weeks. If nobody complains, you have your answer. Then compare its cost against the thing it replaced rather than against zero, check that every benefit number traces to a measurement, and ask whether it changed what your team can do or only what that work costs.

17ResonAi builds audited against systems of record

6public claims that did not survive that audit

0paying customers on any of our SaaS products

ResonAi · Vancouver, BC · Updated September 2026

Four questions, in this order

1. Turn it off

The cheapest and most reliable test, and almost nobody runs it. Switch the pilot off for two weeks without announcing it, then see who notices. A tool people rely on produces complaints within days. A tool people were politely tolerating produces silence. Silence is a result, and it is worth more than any usage dashboard, because usage counts logins and this counts need.

2. Compare it to what it replaced, not to zero

The usual pilot review compares the cost of the AI system against nothing, which makes any benefit look like a win. The honest comparison is against the cost of what it displaced, including the licence you are still paying for and the hours nobody logged.

What to count	Where people get it wrong
What the pilot costs	Only counting the model bill. Add hosting, review time, and the person who babysits it.
What it replaced	Counting a licence you did not actually cancel. If the old tool is still being paid for, it was not replaced.
What it made possible	Skipping this entirely. Usually where the real value is, and the hardest to put a number on.

3. Trace every number to a measurement

Take the headline benefit figure and ask one question: was this measured, or was it modelled? A model looks like "20 proposals a month at two hours each equals 40 hours saved". That is arithmetic on an assumption. A measurement looks like "we timed 12 proposals before and 12 after". Both are fine. Confusing them is not.

4. Did it change what is possible, or only what it costs?

Cost reduction is real but small, and usually the weakest reason to keep something. An insurance PMO we worked with had a senior PM sampling closure reports by hand once a year. After automating the reading, review dropped from 30 to 50 hours per cycle to 4 to 8 hours, and they moved to four cycles a year. The hours are nice. The frequency change is the capability. If the only answer is "the same thing, slightly cheaper", it is a tool, and tools get cancelled in the next budget round.

We ran this audit on ourselves, and it hurt

In September 2026 we audited every ResonAi build against the actual files, databases and git history rather than against our own marketing pages.

Seventeen distinct builds with working code or a live backend. Every claim was tagged as sourced, inferred or unverified. Here is what did not survive.

A case study claimed 300 to 500 hours saved. The client's own documents said 30 to 50. The larger figure appeared only in our marketing. Retired.
A pipeline described as seven agents had six. One page said seven, the code said six.
A build described as production-deployed for a client was a demo build. Its own README said "portfolio / demo build, scaffolding complete". Relabelled.
A page said the agent was making dozens of calls a week. The database held 80 agent messages and 10 decision-log rows in total. Rewritten.
Two case pages listed technologies the code does not contain. One listed n8n in a system that is Python and FastAPI. One listed Supabase on a site that makes no Supabase calls.
A described feature did not exist. Intake screening was in the public description and had zero matches anywhere in the source. Removed.

None of it was deliberate. Every one of those numbers felt true when it was written, and drifted from something real one small step at a time. That is the normal failure mode, and it is why the audit has to run against systems of record rather than against your own previous copy.

And the one most firms would leave out

Across those builds, the logged development work came to roughly $16,100 at API list rates. What it actually cost was a consumer subscription. The builds are real and the capability is real.

The commercial validation is not there yet: zero paying customers on any of the SaaS products. We include that because it is the point. A pilot review that cannot say a thing like that out loud is not a review.

The question underneath all four tests Most AI pilots do not stall on model capability. They stall because nobody decided who owns the output when the agent produces it. No owner means no one checks it, no one defends it in a budget review, and no one notices when it quietly stops being used. Decide that before the pilot starts.

Written by Ryan Rezeika, founder of ResonAi, Vancouver, BC. The self-audit was carried out in September 2026 against files on disk, two live Supabase projects, the website git history and local session logs. The $16,100 figure is the API list price equivalent of logged token usage and is an inferred number, not a bill. The closure-report figures come from the client design documents for that engagement.