## Metadata - Author: [[Nick Bostrom]] - Full Title: Superintelligence - Category: #books ## Highlights - The mathematician I. J. Good, who had served as chief statistician in Alan Turing’s code-breaking team in World War II, might have been the first to enunciate the essential aspects of this scenario. In an oft-quoted passage from 1965, he wrote: Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. ([Location 296](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=296)) - In the summer of 1956 at Dartmouth College, ten scientists sharing an interest in neural nets, automata theory, and the study of intelligence convened for a six-week workshop. This Dartmouth Summer Project is often regarded as the cockcrow of artificial intelligence as a field of research. ([Location 315](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=315)) - The ELIZA program showed how a computer could impersonate a Rogerian psychotherapist. ([Location 338](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=338)) - The realization that many AI projects could never make good on their initial promises led to the onset of the first “AI winter”: a period of retrenchment, during which funding decreased and skepticism increased, and AI fell out of fashion. ([Location 357](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=357)) - A new springtime arrived in the early 1980s, when Japan launched its Fifth-Generation Computer Systems Project, a well-funded public–private partnership that aimed to leapfrog the state of the art by developing a massively parallel computing architecture that would serve as a platform for artificial intelligence. This occurred at peak fascination with the Japanese “post-war economic miracle,” a period when Western government and business leaders anxiously sought to divine the formula behind Japan’s economic success in hope of replicating the magic at home. When Japan decided to invest big in AI, several other countries followed suit. ([Location 359](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=359)) - Behind the razzle-dazzle of machine learning and creative problem-solving thus lies a set of mathematically well-specified tradeoffs. The ideal is that of the perfect Bayesian agent, one that makes probabilistically optimal use of available information. This ideal is unattainable because it is too computationally demanding to be implemented in any physical computer (see Box 1). Accordingly, one can view artificial intelligence as a quest to find shortcuts: ways of tractably approximating the Bayesian ideal by sacrificing some optimality or generality while preserving enough to get high performance in the actual domains of interest. ([Location 418](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=418)) - One sympathizes with John McCarthy, who lamented: “As soon as it works, no one calls it AI anymore.”56 ([Location 478](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=478)) - The computer scientist Donald Knuth was struck that “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’—that, somehow, is much harder!”60 Analyzing visual scenes, recognizing objects, or controlling a robot’s behavior as it interacts with a natural environment has proved challenging. ([Location 532](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=532)) - The world population of robots exceeds 10 million. ([Location 555](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=555)) - Evolutionary algorithms, however, require not only variations to select among but also a fitness function to evaluate variants, and this is typically the most computationally expensive component. ([Location 780](https://readwise.io/to_kindle?action=open&asin=B00LOOCGB2&location=780))