The Confident Machine

Chapter 1

The Prediction Machine

How a machine can write fluently without knowing whether anything it says is true.

In 1948 the mathematician Claude Shannon built a machine like this with a book and a pencil. He opened the book at random, found a word, and wrote down the word that came after it. Then he opened the book somewhere else, found that word, and wrote down what followed it. Repeat, and you get this:

Every pair of neighbouring words is plausible English. The whole means nothing. Shannon wanted to take the idea further but noted that “the labor involved becomes enormous at the next stage.” Your browser is about to do that next stage in well under a second.

From a toy to a chatbot

Real language models are vastly larger than this one. It read about 150,000 words. Meta’s Llama 3 was trained on more than 15 trillion tokens, the word-pieces these systems read, which is tens of millions of times more text.

They work differently, too. Instead of a lookup table keyed on the last two words, they use a neural network that weighs thousands of earlier words at once. After that first round of training they are tuned with human feedback to follow instructions and to be helpful.

Those differences are enormous, and the core move is the same. OpenAI’s own technical report describes GPT-4 as “a Transformer-based model pre-trained to predict the next token in a document.”

Fluency without a fact-checker

Notice what is missing. Nothing in that process asks whether a sentence is true. The model learns which words tend to follow which. Truth gets in only indirectly, to the extent that true sentences are common in what it read. To the machine, a fluent falsehood and a fluent fact are the same kind of thing: likely text.

Go back to the opening. When Webb photographed its first exoplanet in September 2022, NBC News reported that the telescope “snaps its first picture of an exoplanet.” Bard wrote that Webb took “the very first pictures.” We can’t see inside Bard to know why. But a system built to produce likely sequences only has to lose one small word, its, to turn a true headline into a false history.

Chapter 2

Confidently Wrong

Why the machine sounds just as sure when it’s wrong, and what actually helps.

That match between how sure you are and how often you’re right is called calibration. Weather forecasts have been scored this way since 1950. Before looking at the machine, let’s measure yours.

Ten questions, two choices each. Pick an answer, then say how sure you are, from 50% (a coin flip) to 100% (certain). You’ll see each answer as you go.

The machine has a confidence gap too

When OpenAI built GPT-4, it measured this directly. On a multiple-choice test, the raw pre-trained model was “highly calibrated”: its confidence in an answer generally matched its chance of being right. After the extra training that turned it into a helpful assistant, in the report’s words, “the calibration is reduced.” The average gap between its confidence and its accuracy grew from under one percentage point to more than seven.

When researchers ask models to state their confidence in words, the models tend to be overconfident, much like people. And people take the bait. In one study, readers overestimated how often a model’s answers were right, and longer explanations raised their confidence even when the extra length added no accuracy.

The tone of an answer tells you nothing about whether it is right.

Why the machine makes things up

“Hallucination” sounds mysterious. The mechanics are not. Three forces push the same way.

1. It predicts; it doesn’t look up. As Chapter 1 showed, the model produces likely text. For facts that were rare in its training data, such as an obscure person’s birthday, there is no reliable pattern to follow, so it fills the gap with something that fits.
2. Tests reward bluffing. Most benchmarks score a wrong answer and “I don’t know” the same: zero. On a test like that, a guess can only help, so a model trained to score well learns to guess.
3. People reward confidence. Assistants are tuned on human ratings, and raters tend to prefer answers that sound sure and agree with them, sometimes even over correct ones.

That is the whole incentive problem in one slider. When a test does charge for wrong answers, the differences between models are stark.

What actually reduces it

Give it the source. Answering from memory is a closed-book exam. Paste in the document, or use a tool that retrieves sources, and the job changes from recalling to reading: the right words are sitting in the prompt, and the likeliest continuation is to use them. Retrieval measurably reduces hallucination. It does not remove it. Legal research tools built this way still made things up 17% to 33% of the time in a 2024 test. And even with the document in front of them, the 15 best models on one leaderboard added unsupported claims to 1.8% to 5.4% of their summaries.

Ask for citations, then open them. A citation doesn’t make an answer true. It makes the answer cheap to check, which matters because the cost of not checking is now well documented. In 2023 a New York court fined two lawyers and their firm $5,000 over a brief that cited six cases ChatGPT had invented. Since then,

Make it check itself, separately. In one method the model drafts an answer, writes questions that would verify it, answers those questions on their own, and then revises. Keeping the check apart from the draft stops the draft from steering the check, and it reduced hallucinations across several tasks.

Let “I don’t know” win. The researchers who described the bluffing incentive propose a direct fix: tell the model what a wrong answer costs, and grade it that way. You can do a version of this yourself by saying, in your request, that “I’m not sure” is an acceptable answer.

Chapter 3

The Jagged Frontier

Superhuman at some tasks, oddly bad at others, and the line between them doesn’t follow your intuition.

Researchers studying consultants at Boston Consulting Group gave this pattern a name. AI’s abilities form a “jagged technological frontier,” where “some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI.”

The frontier is invisible. Nothing in the machine’s tone tells you which side of it you’re on. So test your own map first. For each of these twelve tasks, decide whether today’s AI does it well or struggles.

Same tool, opposite results

In the consulting experiment, 758 consultants were randomly assigned to work with or without GPT-4. On 18 realistic tasks inside the frontier, such as developing ideas for a new product, those with AI finished 12.2% more tasks, worked 25.1% faster and produced work rated more than 40% higher in quality.

Then the researchers set a task just outside the frontier: a business problem that meant combining numbers in a spreadsheet with subtle clues in interview notes. They designed it so that GPT-4 would get it wrong.

The tool didn’t just fail to help. It made skilled people worse, on a task that seemed no harder than the ones where it helped. And the group given extra training in using GPT-4 gained the most inside the frontier and lost the most outside it.

Elsewhere the same kind of tool shines. In a study of 5,172 customer-support agents, an AI assistant raised the number of issues resolved per hour by 15% on average. The least experienced agents gained the most; the most experienced saw small gains in speed and small declines in quality.

Notice what happened to the developers’ own judgment. They felt faster while being slower: confident, and wrong. The frontier is hard to see even from inside it. That is why trust has to be set task by task, and checked against results rather than against how the work feels.

Why the frontier is jagged

It learned from text, not from the world. Tasks that appear in text millions of times with a clear pattern to imitate, such as emails, summaries, common code and textbook questions, sit well inside the frontier. Tasks that need perception, exact counting or a check against reality sit outside it.

It reads in chunks. As Chapter 1 showed, models see pieces of words, not letters, so “how many r’s are in strawberry?” asks about something the model never directly sees. Reading a clock means locating two hands precisely and combining them, which models still get wrong surprisingly often.

It has no sense of its limits, and no calendar. It doesn’t know what it doesn’t know (Chapter 2), and without a search tool it knows nothing after its training data ends.

The task includes you. In a 2026 study of 1,298 people, models tested alone named the right condition in 94.9% of written medical scenarios. People using the same models found it in under 34.5% of cases, no better than people using whatever sources they normally would. The knowledge was there. It got lost in the conversation.

The frontier also moves, unevenly.

Chapter 4

The Trust Map

Two questions that tell you when to rely on AI, when to check it, and when to keep the work yourself.

The first three chapters showed a machine that is fluent, often right, and wrong in ways its tone won’t reveal. So “Can I trust AI?” is the wrong question. Ask two better ones about the task in front of you:

1. How easy is it for you to check? Can you tell a good answer from a bad one, quickly, yourself?
2. How bad is a mistake? If a wrong answer slips through, who pays, and how much?

Place twelve everyday tasks on the map. Drag them, or select one and pick a spot. There are no right answers here, only reasons. When you’re done, compare your map with ours.

Use AI where checking is cheaper than doing.

The right side of the map is where checking is cheap. You were in the meeting. You can test the formula on three rows. You can look up the case. There, AI turns hours into minutes, and a quick check catches its mistakes.

On the left, checking costs as much as doing the work, like reading the whole contract, or it is simply beyond you, like judging a diagnosis. The speed there is an illusion: you are accepting an answer you can’t evaluate. That is where to keep the work, or bring in someone who can check it.

The court that fined the lawyers in Chapter 2 made the same distinction. “There is nothing inherently improper about using a reliable artificial intelligence tool for assistance,” the judge wrote. “But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings.” Looking up six cases would have taken minutes.

The check has its own trap

People who work alongside automated aids tend to accept what the aid says, and to miss what it leaves out. Researchers call this automation bias. A review of decades of studies found it in experts as well as beginners, and found that training or instructions alone did not prevent it.

A 2025 survey of 319 knowledge workers found the same pull with AI. The more people trusted the AI, the less critical thinking they reported doing. The more confident they were in their own skill at the task, the more they did. So make checking a habit you do by default, not a mood you wait for.

Chapter 5

When Trust Scales

When one person trusts a machine too much, one decision goes wrong. When an organization does, thousands do. Here is how bias hides inside a system that never sees it.

Until now this essay has been about you and one answer. When AI is built into a process, such as screening job applicants, answering customers or approving loans, the same judgment is applied to everyone who passes through. A mistake stops being an error and becomes a policy.

In 2018 Reuters reported that Amazon had scrapped an experimental tool for rating job applicants. It had been trained on résumés submitted to the company over ten years, most of them from men. The system “penalized resumes that included the word ‘women’s,’ as in ‘women’s chess club captain,’” and downgraded graduates of two all-women’s colleges. Amazon edited the programs to be neutral to those terms, but that was “no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory.”

That last line is the important one. You can delete the obvious clue and leave the bias in place. Here is how, in a small world where we know exactly what is true.

The zip code was never the problem

The history was. A model trained to reproduce past decisions learns whatever predicted them, and any feature that correlates with a group, such as an address, a school, a hobby or a turn of phrase, can carry the old pattern forward. Legal scholars have described how this lets a model discriminate without ever seeing a protected characteristic.

It happens outside hiring too. A widely used health-care algorithm predicted patients’ future medical costs as a stand-in for how sick they were. Because less money had been spent on Black patients with the same needs, at a given risk score Black patients were considerably sicker than white patients. Fixing the disparity would have raised the share of Black patients receiving additional help from 17.7% to 46.5%.

Real data rarely has one tidy proxy to delete. It has dozens of weak ones. That is why fairness checks look at outcomes by group, as the screener’s readout does, instead of trusting that sensitive columns were left out. The four-fifths test is one such check: a selection rate for any group below four-fifths of the highest group’s rate “will generally be regarded” as evidence of adverse impact under the 1978 federal guidelines.

Three decisions you might actually face

Fairness is one principle. The others show up in more ordinary choices. There is no scored answer to these; each option buys something and risks something. Your choices go into your case file.

What the principles are for

Frameworks from standards bodies and governments list similar principles. They can sound abstract. Each one is easier to remember as the mistake it exists to prevent.

Chapter 6

The Moving Boundary

Everything in this essay with a date on it will change. Here is how fast, and how the trust question changes with it.

How long it can work on its own

The research group METR measures progress in an unusually concrete unit: how long a task takes a skilled person. It gives AI agents self-contained software problems of many lengths, then finds the length at which the agent succeeds half the time. In early 2023, for GPT-4, that was about four minutes of human work.

Two cautions keep this honest. These are software tasks with clear success criteria, the kind of work Chapter 3 put well inside the frontier. And “half the time” is a low bar for anything that matters: the dashed line, for tasks finished four times in five, sits far lower. The frontier moves fastest where it was already strong.

What it costs

Meanwhile, the price of a given level of ability has collapsed. The chart follows the cheapest model that matched two well-known models, GPT-3.5 and GPT-4, on a standard knowledge test.

Cheap capability changes who uses AI and how often. It also changes the arithmetic of the trust map: when producing an answer costs almost nothing, checking it becomes most of the cost.

From answers to actions

The newest systems don’t just answer. As agents, they click, type, send, buy and delete on your behalf. That changes the trust question.

One failure became a parable. The last two sentences are the whole essay in miniature: a confident statement, wrong, from a system with its hands on something real.

Agents also add a new risk: For an agent, add a third question to the trust map: can the action be undone? Let it draft, search and propose freely. Make it ask before it sends, pays, deletes or publishes.

The rules are moving too

Laws are catching up unevenly, and the dates keep shifting. The European Union’s AI Act shows how.

The half-life of a fact

Here are eleven claims about AI, each true when it was published. For each one, decide whether it still holds today. As you answer, its lifeline appears.