Neural networks, explained the way I finally understood them

I read probably five different explanations of neural networks before one actually stuck. Most either stayed at the "it's like a brain!" analogy level forever, or jumped straight into matrix calculus without building up any intuition first. This is my attempt at something in between — the version I wish I'd read first.

Forget the brain analogy for a second

Everyone starts with "neural networks are inspired by the brain," and sure, that's historically true, but I don't think it actually helps you understand what's going on. A more useful way to think about it, at least for me, was this:

A neural network is a big function made out of smaller, simpler functions stacked on top of each other. That's it. Each "neuron" is really just: take some numbers in, multiply each by a weight, add them up, add a bias, then squash the result through a simple non-linear function. Stack a bunch of these in layers, and you've got a network.

Why the "squashing" part actually matters

This part didn't click for me for a while. If every neuron was just doing weighted addition, you could collapse the entire network down into one single linear equation — no matter how many layers you stacked. Depth would be pointless.

The non-linear function (things like ReLU, sigmoid, etc.) is what breaks that. It's what lets the network learn curves, boundaries, and patterns that aren't just straight lines. Without it, you basically just have really expensive linear regression wearing a costume.

So how does it actually learn?

This is the part where "training" comes in, and honestly the intuition here is oddly satisfying once you get it:

1. Feed data through the network, get an output (a prediction).
2. Compare that prediction to the correct answer using a loss function — basically a number that says "how wrong were you."
3. Work backwards through the network figuring out how much each individual weight contributed to that wrongness. This is backpropagation, and it's just the chain rule from calculus applied over and over.
4. Nudge every weight slightly in the direction that would have reduced the error.
5. Repeat this an absurd number of times, on an absurd amount of data.

That's genuinely the whole loop. No magic, no hidden step. It's adjust, check, adjust, check, over and over, at a scale that's hard to intuitively grasp.

What actually made this click

Building a tiny network from scratch — no libraries, just plain code, on something dumb like predicting whether a point is above or below a line — did more for my understanding than any amount of reading. Watching the loss number tick downward, epoch by epoch, and knowing exactly why each weight was moving the way it was, made the whole thing feel real instead of theoretical.

I still don't have great intuition for why certain architectures (transformers especially) work as well as they do at scale — that feels like a different, harder question than "how does a basic network learn at all." Working on that next.