Ben Goertzel

(1) From a dialectic point of view, I think Kurwzeil makes an impressively compelling argument that the rate of progress has been exponential, historically (e.g. using others' milestones).

So it's striking that he doesn't argue for an underlying mechanism, nor for whether it can continue - or even mention it explicitly.

The mechanism seems to be similar to standing on the shoulders of giants - once an improved technology is developed, it can be used for improved search for other technologies.
The search can then be faster, more efficient, can take place in new domains, with greater accuracy - whatever is the the nature of the improvement.

But there's another assumptions: that there will be more to discover, and with a constant density. But why should the frequency of potential discoveries be constant, such that if you seek faster, you'll find faster? To be clear, it does seem to be that way... I'm just wondering if there's an argument as to why it's that way, and why it will continue to be that way as we keep searching. There's the assumption of mediocrity - that we are not at a privileged center of the universe - but is there a better argument?
For example, why shouldn't it be that discoveries become exponentially rarer? So that we have to keep searching faster and faster just to maintain a linear rate of progress.

EDIT: e.g. consider primes, infinite but become less frequent as you go.

(2) I also like Hofstadner's argument that it's not necessarily true that an…

I lost ten years of my life to worrying about AI risk. I try not to be bitter about that as the rationality techniques I learned along the way have helped me immensely. But I do try to caution others who appear to be falling down the same rabbit hole.

It is the AI x-risk crowd that is non-mainstream, so there isn't really an easily accessible collection of opposing viewpoints, because few have bothered to write up their views. The default/null position is that AI, like nearly all other technologies ever, will have a gradual impact on society and will remain subject to human controls. And that's what most people believe, so researchers who don't buy into the x-risk scenario simply dismiss it as they do all crackpot ideas and move on with their lives. (It should be said that Goertzel has written up his response though: http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.html)

"GPT-3 is AGI" is a bit of a strawman, sorry. I don't think anyone believes that. But GPT-3 is a clear demonstration that networks can be trained on todays hardware to have at least enough encoded information as a human mind, and encoded in such a way as to be generically useful. But GPT-3 is not an agent, obviously. In the mind of its creators however, it ought to be relatively simple (read: a few million dollars and a couple of years) to attach the GPT-3 knowledgebase to the kind of reinforcement learning based agent architecture that DeepMind and OpenAI are…

Ben Goertzel, who is a bona fide AI researcher and was connected with SIAI/MIRI before a leadership split, has written about the topic:

http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.html
http://jetpress.org/v22/goertzel-pitt.htm

You might find some other related commentary with the search term "scary idea", which is what Ben has labeled Yudkowsky's side of the FOOM debate typified by the "That Alien Message" story. TL;DR: Goertzel takes friendliness seriously but considers a fast, hard FOOM to be implausible and worrying over it detrimental to the cause.

I don't know of anyone else with qualifications to be counted as a "top AI researcher" that has covered these topics in any significant detail. Those few times I have seen the topic mentioned in interviews (with Pei Wang and Hugo de Garis, for example), it's dismissed out of hand as crazy-talk. You may find some discussion of this at the various AGI conferences though (use search terms like "friendly" or "safe" on the youtube channels).

Sorry Luke. As usual your fearmongering is not going to get far. For example - MIRI skeptic and famed AGI researcher Ben Goertzel signed the letter (as did I) but not because we are primarily concerned with the threat. Rather:

I signed the document because I wanted to signal to Max Tegmark and his colleagues that I am in favor of research aimed at figuring out how to maximize the odds that AGI systems are robust and beneficial.

The letter helps give visibility and to that end better avenues for funding for AGI projects.

[1]http://multiverseaccordingtoben.blogspot.hk/