Luke Muehlhauser

What I've heard is that MIRI has an explicit philosophy of concentrating on the more abstract & theoretical aspects of AI safety. The idea being that if AI safety is something that you can just tack on to a working design at the end, they don't have a comparative advantage there: it's difficult to predict which design will win and the design's implementor is best positioned to tack on the safety bit themselves.

>...imagine a hypothetical computer security expert named Bruce. You tell Bruce that he and his team have just 3 years to modify the latest version of Microsoft Windows so that it can’t be hacked in any way, even by the smartest hackers on Earth. If he fails, Earth will be destroyed because reasons.

>Bruce just stares at you and says, “Well, that’s impossible, so I guess we’re all fucked.”

>The problem, Bruce explains, is that Microsoft Windows was never designed to be anything remotely like “unhackable.” It was designed to be easily useable, and compatible with lots of software, and flexible, and affordable, and just barely secure enough to be marketable, and you can’t just slap on a special Unhackability Module at the last minute.

>To get a system that even has a chance at being robustly unhackable, Bruce explains, you’ve got to design an entirely different hardware + software system that was designed from the ground up to be unhackable. And that system must be designed in an entirely different way than Microsoft Windows is, and no team in the world could do…

Instead of telling you what they recommended, I'm going to give you my modern list instead of assurance activities that could be considered for the next evaluation scheme. Orange Book had a subset of these. I pulled from the whole field to get a larger list.

https://lobste.rs/s/mhqf7p/static_typing_will_not_save_us_from_broken/comments/d7gega#c_d7gega

The two summaries of the history of successes and problems in TCSEC are here:

http://lukemuehlhauser.com/wp-content/uploads/Bell-Looking-Back-Addendum.pdf

https://www.acsac.org/2004/papers/ClassicPaperSchafer.pdf

You can chart outcomes instead of trying to score inventions.

See for example: http://lukemuehlhauser.com/wp-content/uploads/all-curves-with-events.png

I'd caution that these graphs are completely useless for talking about a technological singularity though, and - even worse - they make it seem like "something must happen" because that line is so obviously "almost vertical". This simply isn't the case, and - importantly - is almost the opposite to the argument for a singularity.