There are smart people working on it, see miri: intelligence.org/ But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: twitter.com/elonmusk/status/49575930734…

You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced. (Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. amazon.com/Superintelligence-Dangers-St…... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about. twitter.com/elonmusk/status/49575930734…

Sam Altman in 2015 - "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity" blog.samaltman.com/machine-intelligence… Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" x.com/elonmusk/status/495759307346952192 Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" 80000hours.org/podcast/episodes/the-wor…... They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me. Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.

If you're actually interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom. amazon.com/Superintelligence-Dangers-St…... It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Musk that this is worth worrying about. twitter.com/elonmusk/status/49575930734…

> TL;DR: It's about the "Internet of Things" ^ this is too ambiguous to be meaningful for anyone who isn't familiar with the cyber grand challenge (CGC). the CGC, as i understand it, is about developing systems which can autonomously detect and patch vulnerabilities[0]. i believe this is interesting for a couple of reasons: 1. a self-repairing system could prevent data breaches (or mitigate damage). 2. this type of system could use its knowledge to attack other systems. on a somewhat related note, i'm reminded of elon musk's reaction [1] to prof. nick bostrom's book [2], "superintelligence": "We need to be super careful with AI. Potentially more dangerous than nukes." though i don't claim to know the motivation/context behind musk's remark, it's easy to envision the dangers associated with a couple of AI's "duking it out" on the interwebs. (CBS's "person of interest" is a fun, fictional series that covers this scenario [3].) [0] here are a couple of participating teams: * forallsecure.com/ * trailofbits.com/ [1] twitter.com/elonmusk/status/49575930734… [2] en.wikipedia.org/wiki/Superintelligence…... [3] en.wikipedia.org/wiki/Person_of_Interes…...