We trust people every day, almost never because they are brilliant. We trust them because we know, more or less, what they will do when no one is watching. Trust was never about brilliance. It was always about consistency.
I forgot that for a while, like almost everyone else, in the face of machines that started to seem intelligent. It only took seeing them fail to remember a simple and old distinction: capable and reliable are not the same thing. One is how much something can do. The other is how much you can count on it. We began treating the two as if they were one.
It is worth understanding what these machines are, because they do not know things the way we imagine. They do not keep a truth to give back. With each question, they build a plausible answer from scratch. Ask twice and get two answers. What they produce has the shape of knowledge, and that shape is what deceives.
When people say one of them has become more intelligent, what improved was the middle answer. That is what tests measure and what announcements celebrate. But what sustains or breaks anything serious is not the middle answer. It is the worst one, and how often it appears. Intelligence is about the average. Trust is about the exception.
There is an old law for this, Goodhart's law: when a measure becomes a target, it stops being a good measure. We started aiming at the test, and it stopped telling us what matters.
I spent most of my life building systems that need to give the same answer to the same question, a million times in a row. It is a craft that teaches a simple distrust early on: you do not build anything on a success that does not repeat.
The instinct is to expect that a more powerful machine will settle the matter. In part, it does move things forward. But there is a fact as well documented as it is uncomfortable: making them more useful often makes them more confident when they are wrong. We train them to seem right, not to admit what they do not know. In a person, doubt is a sign of maturity. In them, it has been trained out. They speak with the firmness of someone who knows and the substance of someone who is guessing, and we should not be surprised when they mislead us. They are doing what we asked.
In an isolated question, we can live with that. The problem grows when they begin to drive entire processes, one step inheriting the uncertainty of the previous one. A chain of almost-certain steps ends far from certain, and the error reaches the end dressed as success. That is why so many promises dazzle in the room and fall apart in the world.
What to do about this is older than AI. Whenever something was too important to be left to improvisation, humanity did the same thing. It thought carefully once, and then stopped thinking. Law is deliberated carefully and then simply applied. A blueprint is drawn once and then built. Civilization is taking the brilliant and unpredictable mind out of the path of repetition and turning discovery into something that does not need to be rediscovered every morning. Intelligence is discovery. Trust is what is born when discovery becomes something one no longer needs to think about.
The question keeps changing. For years, we wanted to know how much machines can do. The question that matters now is harder, and it is not solved by more power: how much can they be trusted? It is the same question we have always asked about anything we decide to rely on.
And there is an irony underneath all this. The better the machine, the better the decision it leaves ready, and the stronger the reason to use it once and then take it out of the way. The most intelligent one is not the safest to leave loose. It is the most dangerous, because its errors are now too plausible to be noticed.
In the end, the test of any intelligence, human or artificial, was never its best day in front of whoever is watching. It is what it does on its worst day, when it is alone and no one checks. It was always the hardest question we ask before trusting someone with something important. Now it is the question we need to ask machines before handing them decisions we cannot afford to get wrong: Can I count on you when I am not watching?