Trying to understand how AI learned to see

One thing I've been thinking about lately is how strange it is that I can give an AI a picture and ask, "What's in this?" — and it can usually give me a surprisingly good answer.

Even stranger is the other direction.

I can describe a scene that doesn't exist, and a few seconds later there's an image of it sitting in front of me.

I've used these tools enough that the result doesn't feel particularly magical anymore. But when I actually stopped to think about what has to happen between "here's an image" and "I understand what's in it", or between "draw me a red house in the snow" and the final image, I realized I had a very vague mental model of the whole thing.

So I tried to make that model a little less vague.

First, an image is just numbers

The easiest place to start is probably the least exciting part.

A computer doesn't see a photograph the way I do. It sees numbers.

A digital image is essentially a grid of pixels, and each pixel contains numerical values describing its color. A small RGB image might have millions of these values.

That's useful because neural networks are very good at working with numbers.

But there's an immediate problem: knowing the numbers isn't the same as knowing what they mean.

If I show you a picture of a dog, you don't consciously inspect every pixel and calculate where the edges are. You recognize shapes, textures, patterns, and relationships almost instantly.

An AI model has to learn its own way of doing something vaguely similar.

How does a model learn what's in an image?

The basic idea is surprisingly straightforward: show it an enormous number of examples.

Imagine training with millions or billions of images paired with information about what they contain. Over time, the model starts picking up patterns.

Certain arrangements of pixels tend to look like edges.

Certain edges and shapes tend to form eyes.

Eyes, ears, fur, and particular body shapes tend to occur together in images of dogs.

And eventually the model can build up much more complicated associations.

The important part is that nobody has to manually program all of these rules.

You don't tell the model:

If there are two eyes, four legs, fur, and a tail, then probably dog.

Instead, the model adjusts a huge number of internal parameters while training so that its predictions gradually become better.

What emerges is a complicated mathematical representation of visual concepts.

And this is where I find things particularly interesting.

The model doesn't necessarily have a little picture of a dog stored somewhere inside it. It has learned statistical relationships that allow it to recognize patterns associated with the concept of a dog.

Images become something the model can work with

Modern vision models usually transform an image into a representation that's easier for the neural network to reason about.

One common approach is to break the image into smaller pieces, or patches, and turn those patches into numerical representations called embeddings.

The model can then work with these representations rather than treating the entire image as one enormous block of pixels.

This is somewhat similar to what happens with language models, where text gets broken down into tokens.

The analogy isn't perfect, but I find it useful:

• Text → tokens → numerical representations
• Image → visual patches → numerical representations

Once everything is represented numerically, the same general machinery of neural networks can operate on it.

That's also why modern AI systems can start connecting things that used to live in separate worlds.

A picture can be related to words.

A word can be related to a picture.

A caption can describe an image, and an image can provide evidence for a caption.

The model is learning relationships between these representations.

Then there's the other direction: making an image

This is the part I originally found harder to picture.

If an AI can recognize an image, that's one thing.

But how does it actually create one?

The answer that helped me most was: it doesn't simply draw the picture pixel by pixel from a written description.

A lot of modern image generation is based on a technique called diffusion.

The basic idea is wonderfully weird.

During training, you take real images and gradually add noise to them.

A little noise.

Then more.

Then more.

Eventually, the original image is almost completely destroyed and you're basically left with static.

The model learns to reverse that process.

Given a noisy image, it learns what direction would make that image look a little more like something meaningful.

Do that repeatedly, and you can go from noise toward an actual image.

So where does the prompt come in?

Now imagine combining that process with language.

You type:

a small wooden cabin beside a frozen lake at sunrise

The text is converted into a numerical representation that captures the relationships between those words.

The image model then uses that information to influence the denoising process.

Instead of simply asking, "What image could this noise become?", it's effectively being guided toward an image that matches the concept represented by your prompt.

Step by step, the random noise becomes structured.

Some shapes start appearing.

Then larger forms.

Then details.

Eventually you get something that looks like a cabin, a lake, snow, sunlight, and all the other things you asked for.

The model isn't looking up a photograph of that exact cabin.

It's generating a new arrangement based on patterns it learned during training.

That's the part that still feels slightly unbelievable to me.

Why generated images can look so convincing

The model has seen an enormous variety of visual relationships during training.

It has learned things like how shadows tend to behave, what perspective looks like, how faces are structured, what materials look like, how objects interact with their surroundings, and which visual patterns tend to appear together.

It doesn't learn these as a neat collection of rules.

They're distributed throughout the model's parameters.

So when you ask for something unusual — say, a glass robot sitting in a Victorian kitchen — it can combine concepts that it has learned separately into something that didn't exist in its training data as a complete image.

That's also why these models can be both impressive and occasionally completely wrong.

They're not checking reality.

They're producing something that fits the learned patterns.

And that's probably why hands used to be such a problem

The famous bad AI hands make a lot more sense once I started thinking about the problem this way.

A hand isn't just "five lines attached to an arm."

It's a complicated three-dimensional object with joints, perspective, occlusion, skin, fingernails, different poses, and an enormous number of possible configurations.

If the model hasn't developed a sufficiently strong internal representation of those relationships, it can produce something that looks hand-like without actually being structurally correct.

The same thing happens with text inside images, reflections, unusual perspectives, and objects interacting in physically strange ways.

The model can be very good at producing the visual appearance of something without having the same understanding of the physical world that a person does.

At least, not in the way I instinctively imagine "understanding" to mean.

What I find most interesting

The thing that has changed how I think about image models is realizing that "seeing" and "creating" aren't really two completely separate abilities.

In both cases, the model is learning relationships.

An image contains relationships between colors, shapes, objects, positions, textures, and other visual patterns.

Language contains relationships between words and concepts.

Once those things can be represented in compatible mathematical spaces, you can start moving between them.

• Image → description
• Description → image
• Image → another image
• Text → image → text

It's less like giving a computer eyes and a paintbrush, and more like teaching it a giant mathematical language for describing patterns.

I'm still not sure what "understanding" means here

This is probably the part I'm least settled on.

When an AI looks at a photo and says there's a dog sitting on a couch, is it actually seeing the dog?

When it generates a convincing dog from a sentence, does it actually understand what a dog is?

I don't think there's a simple answer yet, and I'm not sure the words "see" and "understand" mean quite the same thing for a neural network as they do for us.

But I do think the less mysterious version is more interesting anyway.

There isn't a tiny artist inside the model.

There isn't a database of finished pictures that it simply searches through.

There is a huge learned system of mathematical relationships, trained on enormous amounts of data, that has become surprisingly capable of moving between representations of language and visual patterns.

And somehow, after enough training, starting with noise can end with a picture of a cabin beside a frozen lake.

I'm still trying to get my head around how strange that actually is.