MIRI

Stuart Russell (co-author of AI:MA, one of MIRI's research advisors) argues on http://edge.org/conversation/the-myth-of-ai#26015 that AI systems with "the ability to make high-quality decisions" (where "quality refers to the expected outcome utility of actions taken" and the utility function is represented in the system's programmed decision criteria) raises two problems:

"1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.

"2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task."

The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concerns-about-ai/fulltext). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives."

The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free.

The second of these also has some intuitive force;…

The recent survey on Neurosymbolic Programming is a great introduction to the field: https://twitter.com/swarat/status/1474105098401943555 (pardon the Twitter link). I don't think the full version is publicly available, but I suspect you might be able to get access to it if you DM someone involved.

Other than that, here are some names of people working in this field that you might be able to find representative work from:

- Mayur Naik, UPenn (whose group I think were behind Scallop?)
- Swarat Chaudhuri (and maybe also Isil Dillig), UT Austin
- Luc de Raedt, KU Leuven (whose earlier work on Statistical Relational AI is very similar to the Scallop style of neurosymbolic work - I think there's a good textbook available on this)
- Guy van den Broeck, UCLA
- Armando Solar-Lezama, MIT

If you're more interested in the logic side of things then maybe some of MIRI's (https://intelligence.org/) older work may be of interest.

There's also a lot of people who are interested in neuro-symbolic stuff in the wider sense. You can find out more here: http://www.neurosymbolic.org/index.html. If you sign up for the mailing list, there are monthly (or bi-monthly? I can't remember) talks that are open to the public. You can even find recordings of past talks on Armando Solar-Lezama's youtube channel.

Hope this helps!

>neither of which use alarmist terms like "existential threat"

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

https://www.safe.ai/statement-on-ai-risk

Not sure you're making a meaningful distinction here.

- - -

Of course we all have our own heuristics for deciding who's worth paying attention to. Credentials are one heuristic. For example, you could argue that investing in founders like Bill Gates, Mark Zuckerberg, and Steve Wozniak would be a bad idea because none of them had completed a 4-year degree.

In any case, there are a lot of credentialed people who take Eliezer seriously -- see the MIRI team page for example: https://intelligence.org/team/ Most notable would probably be Stuart Russell, coauthor of the most widely used AI textbook (with Peter Norvig), who is a MIRI advisor.

Please don't... every time a thread like this shows up on HN, I end up impulsively buying some of the recommended books. I'm usually very happy about the purchase, though. :).

Humor aside; not sure if favourite of all times, but definitely impacted my thinking a lot:

- "Rationality: From AI to Zombies" - https://intelligence.org/rationality-ai-zombies/ - read this when it was still a bunch of posts by 'Eliezer on LessWrong. It cleaned up my thinking quite a bit, and introduced to couple new ideas from economics, sociology and epistemology.

- SICP, obviously.

- "How to Win Friends and Influence People", and "How to Stop Worrying and Start Living" - from the father of the whole genre of self-help/personal development, and one whose books are still probably the only good ones in this genre, Dale Carnegie. They explain exactly what it says in their titles.

Σ$0.02: Perhaps a way to contribute to the world would be to figure out how to get rid of Facebook: https://stallman.org/facebook.html

Σ$0.04: Mark Zuckerberg mentions talk on the threat of AI as fear mungering, however, https://intelligence.org/ (MIRI) argues that advanced AI is a legitimate existential threat. Note that when MIRI speaks of "AI", they are not talking about fancy neural networks, but the fact that since humans are no where near any kind of upper bound on intelligence, and the space of possible Turing machines and self modifying algorithms is overwhelmingly large, plus reams of other arguments you , humans will eventually be able to produce computer programs with the capability of destroying civilisation.
N.B: This is my brief summary of what their stance seems to be, and isn't a direct quotation or paraphrasing of MIRI's official opinion.

This paper is published during an era where the dominant global empire is founded and (mostly) run on principles opposite to ethnocentrism, at least internally.

All of the current POTUS' grandparents were born in Europe, and the previous one's father was Kenyan. The most famous entrepreneur of our generation was born of Syrian parents. The wealthiest self-made woman in America according to Forbes (I did have to Google it), Marian Ilitch, had Macedonian parents.

The best strategy in the Iterated Prisoner's Dilemma is tit-for-tat [1]. The winning groups of today have found ways to cooperate-cooperate systematically, overriding the advantages conferred by the behaviour observed in the paper.

[1] It gets more interesting when both sides know how the other thinks: https://intelligence.org/files/ProgramEquilibrium.pdf