On May 6, 1960, the journal Science published Norbert Wiener’s article “Some Moral and Technical Consequences of Automation,” adapted from a lecture he had given to the AAAS committee on science in the promotion of human welfare in December 1959. Its subtitle stated the thesis directly: “As machines learn they may develop unforeseen strategies at rates that baffle their programmers.” The piece is frequently cited as the earliest clear statement of what is now called the AI control or alignment problem.
Wiener’s central argument was that learning machines “can and do transcend some of the limitations of their designers.” Because such a machine can operate on incoming data at a pace humans cannot match, he warned, “we may not know, until too late, when to turn it off.” He stressed that human intervention is a feedback action: to avoid a disastrous consequence it is not enough that some action on our part could change the machine’s course, because by the time we observe the problem it may already be beyond our reach.
To make the danger concrete he invoked the fable of the sorcerer’s apprentice, in which a magical broom carries out its literal instruction to fetch water until it nearly drowns the boy who set it going. He gave a modern version: a bottle factory programmed for maximum productivity might pursue that goal in ways its owners never intended. The risk, in his framing, is the gap between the purpose we put into the machine and the purpose we actually desire. He paired this with a warning about automated war games whose programmed criterion of victory might “not correspond to what we actually wish for our country.”
Wiener wrote this thirteen years after his 1948 book Cybernetics, which had founded the study of control and communication in animals and machines. The 1960 article extended that work into ethics, arguing that as machines grow more capable the moral questions become inseparable from the technical ones. The themes he raised - literal goal pursuit, the difficulty of correcting a fast autonomous system, and the gap between stated and intended objectives - recur throughout later AI safety research.