Artificial Intelligence as a Positive and Negative Factor in Global Risk

Artificial Intelligence as a Positive and Negative Factor in Global Risk
Eliezer Yudkowsky
([e-mail]) Forthcoming in Global Catastrophic Risks, eds. Nick Bostrom and Milan Cirkovic Draft of August 31, 2006

Singularity Institute for Artificial Intelligence Palo Alto, CA Introduction By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A curious aspect of the theory of evolution is that everybody thinks he understands it." (Monod 1974.) My father, a physicist, complained about people making up their own theories of physics; he wanted to know why people did not make up their own theories of chemistry. (Answer: They do.) Nonetheless the problem seems to be unusually acute in Artificial Intelligence. The field of AI has a reputation for making huge promises and then failing to deliver on them. Most observers conclude that AI is hard; as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing. The critical inference is not that AI is hard, but that, for some reason, it is very easy for people to think they know far more about Artificial Intelligence than they actually do. In my other chapter for Global Catastrophic Risks, "Cognitive biases potentially affecting judgment of global risks", I opened by remarking that few people would deliberately choose to destroy the world; a scenario in which the Earth is destroyed by mistake is therefore very worrisome. Few people would push a button that they clearly knew would cause a global catastrophe. But if people are liable to confidently believe that the button does something quite different from its actual consequence, that is cause indeed for alarm. It is far more difficult to write about global

Bibliography: Asimov, I. 1942. Runaround. Astounding Science Fiction, March 1942. Barrett, J. L. and Keil, F. 1996. Conceptualizing a non-natural entity: Anthropomorphism in God concepts. Cognitive Psychology, 31: 219-247. Bostrom, N. 1998. How long before superintelligence? Int. Jour. of Future Studies, 2. Bostrom, N. 2001. Existential Risks: Analyzing Human Extinction Scenarios. Journal of Evolution and Technology, 9. Brown, D.E. 1991. Human universals. New York: McGraw-Hill. Crochat, P. and Franklin, D. (2000.) Back-Propagation Neural Network Tutorial. http://ieee.uow.edu.au/ ~daniel/software/libneural/ Deacon, T. 1997. The symbolic species: The co-evolution of language and the brain. New York: Norton. Drexler, K. E. 1992. Nanosystems: Molecular Machinery, Manufacturing, and Computation. New York: Wiley-Interscience. Ekman, P. and Keltner, D. 1997. Universal facial expressions of emotion: an old controversy and new findings. In Nonverbal communication: where nature meets culture, eds. U. Segerstrale and P. Molnar. Mahwah, NJ: Lawrence Erlbaum Associates. Good, I. J. 1965. Speculations Concerning the First Ultraintelligent Machine. Pp. 31-88 in Advances in Computers, vol 6, eds. F. L. Alt and M. Rubinoff. New York: Academic Press. Hayes, J. R. 1981. The complete problem solver. Philadelphia: Franklin Institute Press. Hibbard, B. 2001. Super-intelligent machines. ACM SIGGRAPH Computer Graphics, 35(1). Hibbard, B. 2004. Reinforcement learning as a Context for Integrating AI Research. Presented at the 2004 AAAI Fall Symposium on Achieving Human-Level Intelligence through Integrated Systems and Research. Hofstadter, D. 1979. Gödel, Escher, Bach: An Eternal Golden Braid. New York: Random House Jaynes, E.T. and Bretthorst, G. L. 2003. Probability Theory: The Logic of Science. Cambridge: Cambridge University Press. Jensen, A. R. 1999. The G Factor: the Science of Mental Ability. Psycoloquy, 10(23). MacFie, R. C. 1912. Heredity, Evolution, and Vitalism: Some of the discoveries of modern research into these matters – their trend and significance. New York: William Wood and Company. McCarthy, J., Minsky, M. L., Rochester, N. and Shannon, C. E. 1955. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence. Merkle, R. C. 1989. Large scale analysis of neural structure. Xerox PARC Technical Report CSL-89-10. November, 1989. Merkle, R. C. and Drexler, K. E. 1996. Helical Logic. Nanotechnology, 7: 325-339. Minsky, M. L. 1986. The Society of Mind. New York: Simon and Schuster. Monod, J. L. 1974. On the Molecular Theory of Evolution. New York: Oxford. Moravec, H. 1988. Mind Children: The Future of Robot and Human Intelligence. Cambridge: Harvard University Press. Moravec, H. 1999. Robot: Mere Machine to Transcendent Mind. New York: Oxford University Press. Raymond, E. S. ed. 2003. DWIM. The on-line hacker Jargon File, version 4.4.7, 29 Dec 2003. Rhodes, R. 1986. The Making of the Atomic Bomb. New York: Simon & Schuster. Rice, H. G. 1953. Classes of Recursively Enumerable Sets and Their Decision Problems. Trans. Amer. Math. Soc., 74: 358-366. Russell, S. J. and Norvig, P. Artificial Intelligence: A Modern Approach. Pp. 962-964. New Jersey: Prentice Hall. Sandberg, A. 1999. The Physics of Information Processing Superobjects: Daily Life Among the Jupiter Brains. Journal of Evolution and Technology, 5. Schmidhuber, J. 2003. Goedel machines: self-referential universal problem solvers making provably optimal self-improvements. In Artificial General Intelligence, eds. B. Goertzel and C. Pennachin. Forthcoming. New York: Springer-Verlag. Sober, E. 1984. The nature of selection. Cambridge, MA: MIT Press. Tooby, J. and Cosmides, L. 1992. The psychological foundations of culture. In The adapted mind: Evolutionary psychology and the generation of culture, eds. J. H. Barkow, L. Cosmides and J. Tooby. New York: Oxford University Press. Vinge, V. 1993. The Coming Technological Singularity. Presented at the VISION-21 Symposium, sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute. March, 1993. Wachowski, A. and Wachowski, L. 1999. The Matrix, USA, Warner Bros, 135 min. Weisburg, R. 1986. Creativity, genius and other myths. New York: W.H Freeman. Williams, G. C. 1966. Adaptation and Natural Selection: A critique of some current evolutionary thought. Princeton, NJ: Princeton University Press.