AI Doesn't Understand "That Earlier One"

I recently came across a question in Fei-Fei Li’s memoir, The Worlds I See:

How would you explain red to someone who has never seen it?

My first thought was analogy. You could describe what red looks like through apples, fire, or blood. You could also describe the feelings and associations it carries on a more abstract level: passion, celebration, warmth.

But the more I thought about it, the less that seemed to work. Even if you explained all of those things clearly, the other person would only know a great many concepts about red. They still would not have actually seen red.

I kept following the question until it wandered into metaphysics. The further I went, the more abstract it became, and the harder it was to put into words. I had no answer, so I set it aside for the time being.

While thinking about it, though, I remembered an idea I once heard on a podcast. It was something along the lines of “language is not a container” or “what you say is not what you really mean.” Take this common exchange at work:

The earlier approach was probably better after all…lol.

Someone who has just joined the project may have no idea what that means. Which approach? When did you discuss it? Why should you use it?

A colleague who took part in the last few discussions will understand immediately. They know which conversation “earlier” refers to, and they remember why the other options were dropped. Sometimes you do not even need to say what comes next or who needs to confirm it.

The same sentence can carry a lot of information for one person and say almost nothing to another.

What the sentence leaves out

Programmers might think of a diff.

A diff can be tiny because both sides already have roughly the same base. If the bases do not match, those few changed lines become very hard to understand. Communication between people works a little like that. We live in the same world, use similar language, and accumulate shared experience by working together. A lot can go unsaid because we do not need to start from the beginning every time.

“Let’s use that earlier approach after all” is only a small increment. What makes it meaningful is the “earlier” that both people already know.

This may seem far removed from the question of red, but I think there is a resemblance. A word cannot hold all that much by itself. It works more like a clue, or a hook. The person who hears it fills in the rest from their own experience.

Seen this way, is the new team member’s confusion a little like the “hallucinations” we get when talking to AI?

AI knows a lot, but not how the project got here

Over the past two years of working alongside AI, whether writing code, building products, or editing documents, I have seen how much it knows. Product planning, feasibility analysis, prototyping, UI/UX, implementation: pick almost anything and it can talk about it convincingly. Sometimes I feel as if my only job is to find new ways to agree: yes/go ahead/proceed/ok…

But if I casually say, “Let’s use that earlier approach,” trouble starts.

It does not know whether “earlier” means 3 minutes ago or 3 hours ago. It does not know why we rejected approach A 3 hours ago, or that a perfectly reasonable suggestion was already tried in another session. Even a request like “make the page a little simpler” may mean something different to it than it does to me.

When I say simple, I usually do not mean removing a few buttons. What I have in mind may include the information hierarchy, the density of the page, and whether the design will fall apart when we add more features six months from now. Those preferences did not form in a single sentence. They became specific through repeated discussion and tradeoffs.

If none of that process survives, AI can only guess again from its general understanding of “simple.”

At that point it looks as if it has suddenly become stupid. Its best practices are still best practices, and its ability to reason has not disappeared. It simply missed how the project arrived here, one step at a time.

I used to blame this on a bad prompt or an insufficient context window, then respond by writing a more detailed request. After doing this often enough, I realized that what I kept adding was usually not a new requirement. It was old history.

Which paths we tried. Which approaches we abandoned. What a particular word means inside this particular project. All of it is scattered across old conversations, documents, and my own head. Every new session means piecing it together again. (To be fair, many agent runtimes are now reasonably good at handling context and memory across sessions.)

The burden is small, more annoyance than injury. Still, it is like being nicked by tiny blades over and over. It gets to you.

That is why I no longer think of Context Engineering as simply stuffing in a few more documents. Documents help, of course, but they usually record what something is now. They do not always explain why it became this way. Rejected approaches, preferences shaped through repeated debate, even the tone people take for granted when they talk to one another: all of these affect how AI interprets the sentence in front of it.

I kept turning these thoughts over until, somehow, they led me from product collaboration back to writing.

Where AI’s knowledge of red comes from

But if I follow this line of thought a little further, something still does not quite add up.

AI did not grow up as a person. It has never waited at a red light or pulled its hand away from a flame. Yet if you ask what red is, it can discuss wavelengths, blood, and danger. It knows what red may represent during Lunar New Year, in religion, or in politics. Its answer may be more comprehensive than what most people could give.

It seems a little rash to conclude from this that it either does or does not understand red.

The texts AI reads were written by people. People have seen flames, been burned, and learned what injury feels like, then turned those experiences into language. A model has never lived through any of this, but it has read vast numbers of descriptions left by people who have. From them, it can learn the relationships among flame, heat, pain, and danger.

It is a little like reading a great many travel guides. You may never have visited the place, but that does not mean you know nothing about it. Of course, reading every guide in the world is still different from actually standing there. (The meaning of travel is yet another ineffable thing AI probably does not quite get.)

Seen this way, what AI has is more like a vast and complicated body of secondhand experience.

Public concepts like “red” have been explained and used by countless people, and their meanings have been revised again and again across different situations. A model can slowly assemble a rough structure from all of that material.

“That earlier one,” however, exists only in the history shared by me and the project. It is not in the public record. The model could not have seen it during training, and it may not exist in a new session either. Unless the earlier discussions and decisions are put back into the context, the model has little chance of guessing correctly.

That creates a strange contrast. It can explain why seeing red makes people tense, yet it does not know why I refuse to use the approach we rejected only 3 hours ago.

How do we preserve “earlier”?

If history is what is missing, the obvious solution sounds simple: write it down.

Anyone who has worked on a real project knows it is not that easy. Documentation usually cares more about what was decided than everything that happened along the way. Maintaining a devlog is easy. Maintaining a thinklog is hard. Why was approach A dropped? What constraints led to a particular compromise? After several rounds of discussion, what exactly does “simple” mean here? These details easily scatter across chat logs and the heads of the people involved.

By the time someone new joins, or you start a fresh session or switch agent runtimes, only the result is left. The reason behind that result may already be gone.

Suppose the only note says:

The user likes simple UI.

It looks perfectly reasonable, yet somehow says almost nothing. The next time the page changes, AI still will not know whether “simple” means fewer buttons, a clearer information hierarchy, lower density, or avoiding a short-term change that will damage future extensibility.

If we save only the conclusion, many words soon collapse back into vague, generic meanings. Saving a little of the tradeoff behind the conclusion could spare us a great deal of repeated explanation later.

Of course, that does not mean keeping every chat log forever. Too much information becomes another kind of noise. A judgment from six months ago may already be stale today. An AI that clings to old context would probably be even more frustrating than one that remembers nothing.

So the hard part may not be memory alone. It is also deciding what to keep, when to forget, and how to help the next person or AI understand why a choice made sense at the time. I have not figured out where that boundary should be.

But I do not really want a “super notebook” that remembers everything. I want the project to be able to continue thinking. Someone else can take over, or I can switch models or return months later, without first reconstructing all the context scattered across the floor.

Looking again at “red” and “that earlier one,” they depend on different kinds of background.

We can talk about red because humans have similar senses and bodies, and because we have long lived in roughly the same physical world and cultural environments. That shared experience starts accumulating early, so early that we barely notice it. It is as if each of us completed a very long round of “pretraining.”

“That earlier one” depends on a small patch of local history. A colleague took part in it, so they can fill in the missing pieces without effort. AI did not take part, and the history was not preserved, so naturally it has to guess again.

The next time this happens, I will probably still add more to the prompt and context. But before I write another long explanation, I should check whether the old decisions and tradeoffs exist somewhere the AI can find them.

Otherwise, however detailed the prompt becomes, I am only making another “first meeting” more complete.