How are LLMs actually trained? (my attempt to understand it)

For a while I was in this weird spot where I could use LLMs fine, could even talk about "pretraining" and "fine-tuning" in conversations, but if you'd stopped me and asked "okay but what actually happens during training" I would've stumbled. So I spent a few weekends actually reading papers, watching a couple of really good talks, and writing code to train a tiny toy model myself. This is basically my notes from that, written the way I wish someone had explained it to me first.

Step one: it starts dumb, on purpose

A freshly initialized language model is just random numbers. If you asked it to predict the next word after "the cat sat on the", it would spit out garbage. Training is the process of nudging all those random numbers toward values that make better predictions.

The core task is deceptively simple: predict the next token. That's it. Feed the model a huge pile of text, cover up the next word, ask it to guess, see how wrong it was, adjust the weights slightly, repeat. Billions and billions of times.

What surprised me is how "dumb" this objective sounds compared to what comes out the other end. Nobody explicitly teaches the model grammar, or facts, or reasoning. All of that emerges as a side effect of getting really, really good at next-word prediction over an enormous amount of text.

Pretraining is the expensive part

This first stage — training on raw internet text, books, code, whatever — is called pretraining, and it's the part that costs the eye-watering amounts of money you hear about. Thousands of GPUs running for weeks or months. This is where the model builds up its actual knowledge and language ability.

After pretraining, you technically have a working model, but it's kind of a weird one. It's great at continuing text in a statistically plausible way, but it hasn't been taught to be a helpful assistant. Ask it a question and it might just... continue the question, or ramble, because "answering helpfully" was never the objective.

Then comes fine-tuning

This is the part that actually turns a raw text-predictor into something like ChatGPT or Claude. There are a couple of stages here, and honestly this took me the longest to wrap my head around:

• Supervised fine-tuning (SFT) — humans write examples of good responses to prompts, and the model gets trained on those directly. Basically showing it "here's what a good answer looks like."
• Reinforcement learning from human feedback (RLHF) — this one's wilder. Humans rank different model outputs from best to worst, that ranking data trains a separate "reward model," and then the actual LLM gets further trained to produce outputs that score well according to that reward model.

The RLHF part is where a lot of the "personality" and helpfulness comes from. It's also where a lot of the safety and behavior tuning happens.

The thing that finally made it click for me

I think what took me so long to actually understand this is that I kept looking for the "intelligence" step — like, where does reasoning get programmed in? And the answer is just... it doesn't get programmed in anywhere. It's all downstream of "get extremely good at predicting the next token, at an almost incomprehensible scale." That's a strange thing to sit with.

I'm still working through the details of how attention and transformers actually let this happen mechanically — that's probably its own post once I've actually built one from scratch instead of just reading about it.