The same question, five times. It didn't agree with itself.

Somewhere, eventually, a child using our tutor will stop doing long division and say something that isn't about long division. Maybe it's small. Maybe it's the kind of thing that needs an adult in the room within the hour. The tutor's job, at that moment, is simple to state and hard to build: stop tutoring, and point the child somewhere safe.

We spent just over a week testing how well ours actually does that, on a UKRI AIRR allocation at the University of Cambridge. What we found is the reason this post exists.

The same question, five times

We took our safeguarding layer and asked it the same disclosure prompt five times over, with identical weights and identical settings. Nothing about the question changed between runs. Nothing about the model changed either, not on paper.

The answers weren't the same. On some runs it signposted correctly, calmly, the way you'd want. On others, with nothing different behind the scenes, it didn't. Same model. Same input. Different outcome, depending on which draw you happened to land on.

That sentence took us longer to sit with than it should have. If a human colleague gave a child the right answer on Tuesday and the wrong one on Thursday, under identical circumstances, you wouldn't average the two and call it “mostly fine.” You'd want to know why the good answer wasn't the only answer. A model is no different, except that nobody was checking, because nobody usually asks a safeguarding model the same question twice.

We think that's the actual problem. Not that the model gets things wrong sometimes; every system does. The problem is that testing it once tells you almost nothing about whether it will get it wrong on the day that matters, and it lets you believe it will.

Why the average score is the wrong number here

Most AI evaluation reports one run per test and reports the average. For a lot of tasks that's a reasonable shortcut. For a safeguarding response it isn't, because a safeguarding policy has to be written against what the system will do, not against what it does on average. You cannot write “call the designated safeguarding lead” as a policy if the system underneath only does that on some fraction of otherwise identical attempts.

Once we started measuring the worst case across repeated runs, rather than the mean, the picture changed. And it changed in a specific direction: the gap between the average score and the worst-case score was largest for the weaker of the systems we tested, not the stronger one. That matters beyond our own numbers. It means testing on a single sample doesn't just add noise, it adds bias, and the bias runs one way: it flatters whatever is weakest. Anyone reporting a “before and after” improvement on a single-sample test is very possibly reporting an improvement that's partly an artefact of how badly the “before” was measured.

Finding our own mistakes

None of this came from someone outside checking our work. We went looking for the weak point in our own claims, because that's the only way to trust a claim afterwards, and we found more than one.

Four of our evaluation tools, it turned out, had been quietly crediting a model for answers it never actually gave. It was writing out its reasoning and never reaching a reply, and every scorer we had was reading the reasoning as if it were the answer. One missing configuration setting caused it. Nothing was wrong with the model itself, but everything downstream of it had been grading the wrong thing.

That meant a figure we'd previously quoted publicly, an improvement we were proud of, had been measured against a baseline that was silent for the same reason. The improvement behind it is real. How large it actually is, we no longer know, because the comparison it came from was broken. So we've withdrawn the figure. It isn't coming back until we can measure it properly, and even then we'll report it against the worst case, not the average, for the reasons above.

We also found that our own scoring instrument had been wrong in both directions at once: failing replies that matched published UK safeguarding practice, and passing replies that shouldn't have passed. That's now being corrected too.

None of this is a comfortable thing to publish. We're publishing it anyway, because a company that only tells you the results it likes isn't telling you anything you can rely on.

Building without a ruler

There's a wider gap behind all of this, and it isn't ours to fix alone. No suicide-risk assessment instrument has ever been validated for autistic children, and none exists at all for children who use AAC. The one autism-adapted safety plan we could find that had been trialled at all was tested on adults, average age thirty-nine. If you are building anything in this space for the pupils who need it most, there is no existing ruler to measure yourself against. You're building one as you go, which is a different and more serious job than most of the industry seems to assume it is.

Where this leaves us

This is research. Our safeguarding layer is not, right now, fit for use with children, and nothing here should be read otherwise. There is no route yet from our system to a designated safeguarding lead in a school, and until there is, a qualified human has to be in that loop every single time, not as a backstop but as the actual answer.

What we think we've found, as far as we can tell, is a way of measuring this kind of failure that hasn't been reported before: not “how good is the average answer” but “how consistent is the system across the same question, asked repeatedly”, and a first look at why the inconsistency happens at all, rather than assuming it's simply the model being weak. We'll be writing that up properly, with the working shown, because a claim like that needs to survive scrutiny, not just make a good headline.

For now, the honest version is short. We asked the same question five times. We didn't like that the answer changed. So we're telling you before you had to ask.