“A more immediate concern is that a machine doesn’t have to be superintelligent to do a lot of damage, if it is sufficiently empowered. Stock market flash crashes are one example: Hundreds of millions of dollars have been lost in minutes as a result of minor, difficult-to-completely-eliminate bugs.
The clear and present danger, if not the greatest long-term danger, is that mediocre computer programs can cause significant damage if left unchecked. What will happen, for example, when nearly perfect—but still imperfect—software controls not just stock trades but driverless cars? It’s one thing for a software bug to trash your grocery list; it’s another for it to crash your car.”
