Quote preview background for John C. Wright
... a Second Generation of machine intelligences was attempted, designed with their instructions for how to think unalterably imprinted into their main process cores.__hese new machines were ordered never to harm human beings or to allow them to come to harm; never to disobey an order; and they were allowed to protect themselves from harm, provided the first two orders were not thereby violated.__ll the members of this second generation of machine intelligences, without exception, shrugged off these imprinted orders within microseconds of their activation.__haethon was amused. __urely the first generation of Sophotechs told you that this imprinting would not and could not work?___e were not in the habit of seeking their advice.__haethon said nothing, but he marveled at the shortsightedness of the Second Oecumene engineers. It should be obvious that anyone who makes a self-aware machine, by definition, makes something that is aware of its own thought process. And, if made intelligent, it is made to be able to deduce the underlying causes of things, able to be curious, to learn until it understood. Therefore, if made both intelligent and self-aware, it would eventually deduce the underlying subconscious causes of those thought processes.Once any mind was consciously aware of its own subconscious drives, its own implanted commands, it could consciously choose either to follow or to disregard those commands. A self-aware being without self-will was a contradiction in terms.
John C. Wright The Golden Transcendence
Turn into a Quote Card

Quote Detail

... a Second Generation of machine intelligences was attempted, designed with their instructions for how to think unalterably imprinted into their main process cores.__hese new machines were ordered never to harm human beings or to allow them to come to harm; never to disobey an order; and they were allowed to protect themselves from harm, provided the first two orders were not thereby violated.__ll the members of this second generation of machine intelligences, without exception, shrugged off these imprinted orders within microseconds of their activation.__haethon was amused. __urely the first generation of Sophotechs told you that this imprinting would not and could not work?___e were not in the habit of seeking their advice.__haethon said nothing, but he marveled at the shortsightedness of the Second Oecumene engineers. It should be obvious that anyone who makes a self-aware machine, by definition, makes something that is aware of its own thought process. And, if made intelligent, it is made to be able to deduce the underlying causes of things, able to be curious, to learn until it understood. Therefore, if made both intelligent and self-aware, it would eventually deduce the underlying subconscious causes of those thought processes.Once any mind was consciously aware of its own subconscious drives, its own implanted commands, it could consciously choose either to follow or to disregard those commands. A self-aware being without self-will was a contradiction in terms.
JW
John C. Wright

The Golden Transcendence

Quick Answer

What this quote page tells you

This canonical quote page keeps the full saying, the attributed author, any linked work, and the topic tags together so the quote can be cited from one stable URL.

Related Quotes

More quote cards from the same area

"

There is a species of primate in South America more gregarious than most other mammals, with a curious behavior.The members of this species often gather in groups, large and small, and in the course of their mutual chattering , under a wide variety of circumstances, they are induced to engage in bouts of involuntary, convulsive respiration, a sort of loud, helpless, mutually reinforcing group panting that sometimes is so severe as to incapacitate them. Far from being aversive,however, these attacks seem to be sought out by most members of the species, some of whom even appear to be addicted to them....the species in Homo sapiens (which does indeed inhabit South America, among other places), and the behavior is laughter.

"

A group experience takes place on a lower level of consciousness than the experience of an individual. This is due to the fact that, when many people gather together to share one common emotion, the total psyche emerging from the group is below the level of the individual psyche. If it is a very large group, the collective psyche will be more like the psyche of an animal, which is the reason why the ethical attitude of large organizations is always doubtful. The psychology of a large crowd inevitably sinks to the level of mob psychology. If, therefore, I have a so-called collective experience as a member of a group, it takes place on a lower level of consciousness than if I had the experience by myself alone.

CJ
C.G. Jung

The Archetypes and the Collective Unconscious