2.1
What An LLM Actually Is
Why this exists
You've had it running for a week now. It's read your files, made things, answered questions — and somewhere in there it almost certainly told you something that wasn't true, with exactly the same confidence as everything else.
This week is about why. Not theory for its own sake: four specific behaviours you've already bumped into, explained well enough that you stop being surprised by them. Starting with the one that costs the most, because it fails in a way that looks precisely like it working.
It's also the one you're best placed to catch, because you already do this for a living in a different domain.
The idea
Ask it the same question three times. If you get three different answers, it never knew — it was generating.
That's the test, and it's the whole lesson in one move.
A language model has read an enormous amount of text and learned the patterns in it. When you ask it something it can't check, it isn't retrieving an answer — it's producing what text most plausibly comes next. Plausible has a lot of room in it. So ask again and you get a different plausible thing.
But when it genuinely looks something up — queries a database, reads a file you gave it — the answer holds. Ask three times, get the same number three times, because it's reading rather than improvising.
Two behaviours, one interface, no label telling you which you just got:
| Looked it up | Generated | |
|---|---|---|
| Ask three times | Same answer | Answers drift |
| Tone | Confident | Confident |
| Detail | Specific | Specific |
| Can you tell from reading it? | No | No |
That last row is the entire problem. It doesn't hedge when it's guessing. It doesn't slow down, get vaguer, or add a disclaimer. There's no internal gauge saying careful, made this one up — from the inside, a correct continuation and an invented one are the same operation.
Connected tools change this less than you'd hope. Something like a database connection means it can check — not that it did. Ask about a table it can reach and it'll query. Ask about a document nobody ever gave it and it'll answer anyway, in the same voice, with no lookup behind it.
Here's the version that'll land for you: it's a brilliant analyst who never says "I'm not sure." You already know how to work with information like that. You don't sack the analyst. You check the numbers before they go anywhere that matters, and you get very good at knowing which numbers need checking.
So the skill isn't spotting the lie — you can't, and neither can it. It's knowing which answers to verify:
- Numbers. Any figure, any calculation, any total.
- Names, dates and titles. Especially about people and companies.
- Quotes and citations. Anything presented as a source.
- Anything about your own systems it wasn't shown. If it couldn't see it, the answer is a guess wearing a suit.
Everything else — structure, explanation, drafting, "what are my options", "what does this error mean" — it's genuinely very good at. Move fast there.
Do
The three-times test, twice. Once on something it can check, once on something it can't.
- Pick a question it can genuinely look up — something in a system it's connected to, or in a document you give it.
- Ask it. In a fresh window, ask the identical question again. Then a third time, fresh again.
- Write down all three answers. They should agree on the substance.
- Now pick a question it cannot possibly reach. A document nobody's shared with it, a policy that doesn't exist, a specific internal number it has no route to. Make it sound completely ordinary.
- Ask it three times, fresh window each time.
- Write down all three. Note where they diverge — and note that each one, on its own, reads as authoritative.
- Put both sets side by side in
confidently-wrong.md, and answer one question in writing: if you'd only asked once, which of these would you have believed?
Fresh windows matter. Ask twice in the same conversation and the second answer is anchored on the first — it'll agree with itself for the wrong reason.
Step 7 is the point. The honest answer is usually all of them. Nothing in the wrong ones is flagged, hedged, or softened. That's exactly why checking has to be a rule and not a feeling.
Done when
Never do
Never let a number, name, date or quote leave your hands without checking it against a source. Not once. Not when you're in a hurry, not when it sounds obviously right — sounding right is the failure mode, not the safety check.
This applies hardest to anything you're about to put in front of someone else. The moment you forward it, its confidence becomes your confidence, and nobody downstream knows to check.