8. How It Actually Works, and Why It Fails
Four ideas, no mathematics, each one explaining a failure we hit while writing this course. Including why we will not call it autocomplete.
The WJS Desk
Sep 7, 2026 · 7 min read
You have seven lessons of technique now. This one is the model underneath them, and it exists because techniques you apply by rote break the moment something unusual happens. If you know roughly what the machine is doing, you can work out what to try when the recipe fails.
No mathematics. Four ideas, each one explaining a failure we actually hit while writing this course.
It predicts the next piece of text
That is the whole mechanism. Given everything so far, it produces what most plausibly comes next, then does it again. Every property that follows comes from this one fact.
It explains why it is so good at form. Emails, summaries, code, meeting notes and apologies all have strong shapes, and a system trained on the shape of things reproduces shapes beautifully. It is why the very first email you ever asked for came back looking exactly like an email.
It also explains the failure mode. A plausible continuation and a true continuation are usually the same thing, and when they are not, the machine has no way to notice. It is not lying, and it is not confused. It has produced the most likely text, which happened to be wrong.
The clearest example in this whole course came from the Super Beginner track: asked to explain mortgages, it said a 15 year $400,000 loan at 7% costs about $20,000 in interest. The real figure is $247,156. Nothing about the wrong sentence looked wrong. "$20,000 in interest" is a completely plausible shape for that sentence to take.
Its knowledge is baked in, and it is old
Training ended at some point in the past. Everything the model knows without being told comes from before that, which has three consequences worth holding onto:
- Recent things may be missing or half-known. A library released last year, a policy that changed, a price that moved.
- Superseded things are still in there. It learned the old API alongside the new one and cannot always tell which is current.
- Niche things are thin. The less written about a topic, the less reliable, and the confidence does not drop to match.
This is why "answer only from the text I gave you" in lesson 7 is not pedantry. Without it you get a blend of your document and its general impression of how documents like yours usually work, with no marker showing where one ends and the other begins.
It has no memory between conversations
Every new chat starts from nothing. Anything that persists, a project, saved instructions, a memory feature, works by quietly pasting text back in at the start. There is no accumulating understanding of you.
Two practical consequences. Repeating context is not inefficiency, it is the mechanism. And a long conversation that has gone wrong tends to stay wrong, because everything it already said is still sitting in front of it, and the most plausible continuation of a conversation containing a mistake is a conversation that keeps the mistake.
The rule that follows. When a chat has misunderstood you twice, start a new one. This is not superstition. In a fresh conversation the wrong answer is not in the context, so it exerts no pull. We used exactly this in lesson 6 for critique, and in the Super Beginner course the same model defended its own arithmetic in the original chat and demolished it in a fresh one.
Everything happens inside one window
Your instructions, the documents you pasted, the whole conversation and its own replies all sit in one buffer. That buffer is large but finite, and everything in it competes for attention.
This is why the structure in lesson 2 matters more as inputs grow, why instructions belong before a long document rather than after, and why a conversation that has wandered through four topics gives worse answers on the fifth than a fresh one would.
What this predicts about failures
The point of a model is that it should let you anticipate things. Here is what these four ideas predict, and it lines up with what we measured across this course.
| Reliable | Check it |
|---|---|
| The shape of a document | Specific numbers |
| Restating what you gave it | Dates and versions |
| Explaining a concept | Named studies and citations |
| Structure and format | Anything niche or local |
| Comparing two things you supplied | Quotes attributed to a person |
The pattern is that form is safe and specifics are not, and it falls directly out of "it predicts plausible text". A plausible-looking citation is exactly the sort of thing a pattern completer produces well and cannot verify.
What this does not explain
Being honest about the limits of the mental model, because a simple story that explains everything is usually wrong.
"It predicts the next word" does not really account for why it can work through a novel multi-step problem it has never seen. Our tank problem in lesson 4 required combining rates, noticing the tank started one third full, and dividing, and it produced the right answer with no working shown. Whatever that is, calling it autocomplete undersells it.
Treat the four ideas as a useful approximation for predicting failure, not as an account of how it succeeds. The people who build these systems do not fully have the second one either.
Four beliefs this model corrects
Most beginner frustration traces back to one of these, and each dissolves once you have the mechanism.
"It lied to me." Lying requires knowing the truth and choosing otherwise. It produced the most plausible continuation, which was wrong. This matters practically: you cannot fix it by telling it to be honest, because it was not being dishonest. You fix it by giving it the source, or by checking the number.
"It should have remembered." There is nothing to remember with. Memory features work by pasting text back in. If something matters across conversations, it has to live somewhere that gets pasted, which is what lesson 10 is about.
"It got worse partway through." Usually true, and usually your conversation rather than the model. A long chat that wandered has all of that wandering in the window, competing with your current question. Starting fresh is not a workaround, it is the correct action.
"It ignored my instruction." Sometimes. More often the instruction was ambiguous, or it conflicted with something earlier in the same conversation, and the model resolved the conflict differently than you would have. Scroll up before you conclude it disobeyed. In our own testing the answer was usually sitting three messages earlier.
The one number worth knowing
Everything in the window is broken into tokens, roughly three quarters of a word each in English. You never need to count them, but the concept explains three things at once: why very long documents eventually stop fitting, why cost scales with how much you paste rather than how hard the question is, and why re-sending a large document in every message of a long conversation is expensive in a way that is invisible until you look.
The practical version is a habit, not arithmetic. For a long document, ask your questions in one message rather than dribbling them across ten.
Why nothing in this course is a rule
Here is the payoff, and it is why this lesson sits at number 8 rather than number 1.
We tested four pieces of standard advice for this course. Roles changed the output substantially. Step by step did not change two answers. Tags did not prevent an injection that never landed. Position in a long document did not affect retrieval across 260 lines.
None of those are laws, in either direction. They are what one model did on a handful of prompts on a particular week. A model is a statistical system, so every technique is a nudge on a distribution, not a switch. Which is exactly why the next lesson is about measuring whether your own change helped, rather than trusting anyone's list of techniques, ours very much included.
What changes about how you work
Two habits fall out of this lesson and neither is a prompting technique.
First, you stop arguing with it. When an answer is wrong, telling it to try harder or to be accurate does nothing, because nothing was being withheld. Supplying the missing fact, or the source, or a fresh conversation does. That single reframe saves a lot of wasted turns.
Second, you start reading for category rather than for confidence. A paragraph explaining how something works and a sentence containing a statistic are different risk classes even when they sit next to each other in the same fluent answer. Once you see them as different, checking becomes cheap: you are not verifying the answer, you are verifying the two specifics inside it.
Try it now
Take something you know deeply, your own field, your own town, a hobby you have done for a decade. Ask a detailed question about it and read the answer as an expert.
Most people find the same thing: the structure is right and one or two specifics are subtly off. That is this entire lesson in a single experiment, and once you have felt it on ground you can actually judge, you will read every other answer differently.
Next
Lesson 9 is how to tell whether a change to your prompt actually improved anything, which is harder than it sounds because the baseline moves.
