June 10, 2026

What ChatGPT Actually Can't Do (A Reality Check)

I want to tell you something that most people in technology will never say out loud.

ChatGPT is impressive. It is also profoundly, structurally limited in ways that the breathless press releases and the doom-laden content online both manage to completely ignore.

The hype machine tells you ChatGPT is about to replace lawyers, doctors, teachers, and therapists. The doom machine tells you it is going to destroy truth, democracy, and civilization. Both camps are so busy dramatizing that neither has sat down to honestly describe what the thing actually cannot do.

So let me do that.

It Cannot Know What It Does Not Know

This is the limitation that experts call the problem of calibration. ChatGPT does not know what it knows and what it does not know. It cannot feel the difference between a fact and a fabrication. Both feel identical to it from the inside.

When you ask it a question it cannot answer, it does not say “I do not know.” It says something that sounds like an answer. It fills the gap with plausible-sounding language. It invents citations. It fabricates quotes. It generates court cases that do not exist, medical studies that were never conducted, and historical events that never happened.

A human expert who does not know something pauses. They say “let me check” or “I am not sure.” That pause is not a weakness. It is how knowledge actually works. ChatGPT has no pause. It has only forward motion.

This is not a bug they can fix. It is a feature of how the system is built. Language models predict the next most likely word. They do not verify truth. They simulate fluency.

It Cannot Understand. It Can Only Predict.

There is a famous thought experiment in philosophy called the Chinese Room, proposed by John Searle. Imagine a person locked in a room with a rulebook. They receive Chinese symbols through a slot and use the rulebook to produce the correct response in Chinese. From outside, the room appears to understand Chinese. The person inside understands nothing.

ChatGPT is a very sophisticated Chinese Room.

It processes patterns in text. It has been trained on an enormous amount of human language and has learned that certain sequences of words tend to follow certain other sequences of words. When it writes a paragraph about grief, it is not drawing on any experience of loss. It is drawing on the statistical patterns of how humans have written about grief.

This matters enormously in situations where genuine understanding is the entire point. A therapist does not just produce correct sentences about pain. They notice the moment a client flinches. They adjust in real time. They bring their own history of having suffered and recovered. They offer not information but presence.

ChatGPT can produce sentences about presence. It cannot provide it.

It Cannot Have Skin in the Game

Nassim Taleb built an entire philosophy around the concept of skin in the game. The idea is simple. Advice from someone who bears no consequences for being wrong is fundamentally different from advice from someone who does. A doctor who would have to live with the outcome of their diagnosis approaches medicine differently from one who does not.

ChatGPT has no skin in the game. Ever.

If it gives you bad legal advice, nothing happens to it. If its business strategy destroys your company, it does not notice. If its medical suggestions harm your child, it will cheerfully help the next user. There are no stakes. There is no accountability. There is no weight behind the words because the words cost nothing to produce.

When a good lawyer advises you, they are staking their reputation on that advice. When a great mentor guides you, they are investing something real. That investment changes the quality of what they give you. It forces care. ChatGPT is structurally incapable of care.

It Cannot Think in Genuinely New Ways

ChatGPT is extraordinarily good at recombination. It can take existing ideas and blend, rephrase, reframe, and remix them in ways that feel fresh. This is genuinely impressive and genuinely useful.

What it cannot do is generate the kind of genuinely discontinuous thinking that has driven the most important advances in human history. Darwin did not recombine existing ideas. He saw something that nobody had seen before. Einstein did not remix physics. He broke the frame entirely. Toni Morrison did not produce optimized genre fiction. She invented a voice.

Truly new ideas require being embedded in the world. They require having a problem that actually matters to you. They require failure, frustration, the long slow fermentation of years of thinking about something you cannot let go of. ChatGPT has not lived. It has read about living.

There is a difference between a map and a territory. ChatGPT has consumed an enormous amount of maps. It has never set foot in the territory.

It Cannot Be Trusted With High Stakes Decisions

In early 2023, a lawyer named Steven Schwartz submitted a legal brief in a US federal court. The brief cited six precedents. Every single one of them was made up by ChatGPT. None of the cases existed. The judge was not amused.

This is not an isolated incident. It is a demonstration of the fundamental problem. ChatGPT sounds authoritative regardless of whether it is right. It uses the same confident, fluent tone to describe a verified fact and a complete fabrication. It does not know which is which. You cannot tell from the output.

For tasks where being wrong by ten percent is fine, this is manageable. For tasks where being wrong by one percent can ruin a life, this is catastrophic: Medical diagnosis, Legal strategy, Financial planning, and Structural engineering. These are not domains where you can afford a confident-sounding fabricator.

It Cannot Read the Room

Most of human communication is not in the words. It is in the pause before the words. The slight shift in posture. The catch in the voice. The moment someone looks away. The way a question is phrased that tells you this person is not actually asking what they appear to be asking.

Great teachers, managers, therapists, salespeople, and leaders navigate this constantly. They are not just processing the content of what is being said. They are processing everything that surrounds it.

ChatGPT reads text. It cannot read a room. It has no access to tone, body language, relational history, emotional context, or the thousand small signals that make human communication what it is.

It Cannot Learn From Its Own Mistakes

Every conversation with ChatGPT begins from zero. It does not remember that it told you something incorrect last Tuesday and has thought about why since then. It does not carry the scar of past failures into future interactions. It does not build the kind of wisdom that only comes from getting things badly wrong and sitting with the consequences.

Human expertise is built from accumulated failure. The surgeon who once lost a patient is a different surgeon afterward. The entrepreneur who once ran a company into the ground carries knowledge in their body that no textbook can replicate. Wisdom is failure that has been digested.

ChatGPT has no failures of its own. It has only the described failures of others. There is a vast difference between those two things.

So What Is It Actually Good For?

This is not an argument against using ChatGPT. Many people I know use it. It is genuinely useful for a specific category of tasks.

It is extraordinary at tasks where speed matters more than precision. For instance, first drafts, brainstorming, or summarizing text you already understand. Generating options you will then filter through your own judgment; explaining concepts in plain language, or even doing the boring work of structuring information you already have.

The critical word in that list is “you.” You summarize text you already understand. You filter options through your judgment. You bring the expertise that turns ChatGPT’s raw output into something trustworthy.

ChatGPT is an excellent assistant for people who already know what they are doing. It is a dangerous substitute for people who do not.

The Question Nobody Is Asking

Here is what I find most troubling about the current moment.

We are deploying ChatGPT in exactly the domains where its limitations are most dangerous. Customer service, where people call because they are distressed and need to feel heard. Healthcare triage, where the stakes of a wrong answer are a human life. Education, where the entire point is for a real human mind to struggle against difficulty and grow stronger in the struggle.

We are not asking: what does this tool actually do well? We are asking: where can we use this tool to cut costs?

Those are not the same question. The second question is the one that gets people hurt.

What ChatGPT reveals about us

The most interesting thing about ChatGPT is not what it tells us about machines. It is what it tells us about humans.

When a tool that has no understanding, no accountability, no embodied experience, and no skin in the game can produce output that is genuinely difficult to distinguish from expert human work; it forces us to ask: what exactly is expertise? What is intelligence? What is the thing that humans do that is worth preserving and protecting?

The answer is not in the output. It never was. The value of a human expert is not in the words they produce. It is in the judgment, the accountability, the accumulated experience of failure, the presence in the room, the weight of having something real at stake.

ChatGPT can write faster than any human. It cannot be accountable the way any human can.

In a world that is increasingly drowning in content, accountability is the rarest and most valuable thing there is.

Remember that the next time someone tells you ChatGPT is coming for your job.

Originally published on Substack. ← Back to all articles