You type some words in English into Claude and it writes some back. It looks like the model speaks your language.
Inside the model, text is represented as numbers. What you type in gets converted into numbers, then the model processes those numbers, and the result gets converted back into words for you.
How does it get anything done when all it ever sees is numbers? Say I hand you a wine review written in Japanese, and you only know English. You can't read a word of it, but after a few hundred reviews, you will start to notice things. Some characters always come in pairs. Some show up near the end. Some never sit next to each other. Give it a year or two, or ten (depending on how high your IQ is), and you could guess the next character fairly well, without ever learning what any of them mean.
A model is trained to guess what comes next, and meaning appears in the patterns it learns. That's why it's called a language model: it models the patterns in language.
Turning your words into numbers takes two steps, and each has a name you'll hear a lot: tokenisation and embedding.
How do you chop text into pieces?
McDonald's know what people order together. Fries and a drink go together. A Big Mac nearly always goes with those two. So now there's a Big Mac meal (Big Mac, fries and drink), one thing on the menu instead of three. The same happened with nuggets. An apple pie never settled next to anything in particular, so it's still sold on its own.
Now someone orders a Big Mac, fries, a drink, nuggets and an apple pie. There's no meal for that exact order, so the till rings it up with what it has: a Big Mac meal, plus nuggets, plus an apple pie.
Letters work the same way. Each letter is one item on the menu. A program reads a mountain of text and tallies which letters keep landing next to each other. In English, "t" and "h" sit next to each other more than any other two letters, so they become a pack: "th". That pack and "e" are together more than almost anything else, so they become "the".
When you type "Spaghettify", a word you won't find in most dictionaries and that the program may never have seen, there's no pack for it. You get it in the packs that do exist: Sp, ag, he, tt, ify. Five things instead of eleven letters, like the odd order at McDonald's. Here's that happening as you type:
Common words come as one pack. Rare words, typos, slang and even emoji come in smaller packs. Nothing breaks, because the tiniest pieces (often bytes) are always still on the menu.
The packs are called tokens. The full menu of them is called the vocabulary. The tallying that built it is called byte pair encoding (BPE).
So where do the numbers come in?
Every pack on the menu has a number on it. The exact chunks and IDs depend on the tokeniser and the model. To illustrate, if you type "Spaghettify", its five packs come out as Sp 4916, ag 348, he 273, tt 1037, ify 1985. That number is the token ID.
An ID is a name tag, nothing more. Pack 4916 is no more like pack 1037 than house number 12 is like house number 13. So far the model has a row of name tags, and no idea what any of them mean.
How does the model know which tokens are alike?
Say you work at a supermarket and only ever see receipts, never shoppers. One receipt says nappies, wipes, baby food. Another says nappies, wipes, formula. You'd bet those two have a lot in common. A third says cat litter and tuna. Different shopper.
The model reads words the way you read those receipts. It never sees what "king" means, only that "king" keeps turning up next to thrones and castles, and so does "queen". "Dog" turns up next to tails and walks.
Now imagine pinning each shopper on a map, with shoppers who buy the same things close together. The two baby-food shoppers end up side by side. The cat owner ends up across the map. That's what the model does with tokens: the ones that keep the same company are put near each other. Nobody places them by hand. Every sentence the model reads nudges the words that fit it a little closer, like this:
That spot on the map is called an embedding. It's written as a list of numbers, the way a spot on a paper map is two numbers, only much longer. For the model, meaning is where you sit and who sits near you.
If every "king" gets the same spot, how does the model know which king I mean?
It doesn't yet. The spot is a starting point. The model also notes where each token sits in a sentence (first, second, third and so on), which is called positional encoding. Then it looks back at the earlier tokens to work out what this "king" means here.
Putting it together
Text comes in. It gets handed over in packs, each pack has a name tag, and each name tag has a spot on the map. From there, the model guesses what comes next by looking at who sits near whom.
You saw words at the start and words at the end. In between, the model saw only numbers. It speaks fluently, just not your language.
