Arram Sabeti

Being the author of The Program, a fiction podcast described by fans as "a love child of Black Mirror and NPR", I've been following news around GPT-3 very closely.

So I decided to make an episode depicting a fictionalized origin story of GPT-3. The twist is that I incorporated actual text generated by GPT-3 to tell the narrative (even the episode title has been generated by it!)

To this end, I used four excellent articles and wonderful HN discussions that ensued as sources of inspiration and insight. Some of the comments have made it into the episode almost verbatim (of course, all have been properly credited in the show's references).

I'm talking about:

https://tinkeredthinking.com/index.php?id=841 - discussion: https://news.ycombinator.com/item?id=23923799

https://www.gwern.net/newsletter/2020/05#gpt-3 - discussion: https://news.ycombinator.com/item?id=23623845

https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html - discussion: https://news.ycombinator.com/item?id=23887637

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts - discussion: https://news.ycombinator.com/item?id=23989215

Additionally, I partnered up with a startup called Narration Box to synthesize the computer voice of GPT-3, which I connected with back when I originally unveiled The Program on HN (you might be glad to hear the show recently passed 100,000 downloads).

This community has been nothing but good to me so I thought you might be interested to check out the latest work of whimsy it…

Hmm. Holding one category steady and checking the others invariably reveals a combination that hasn't been generated (like in dozens of tries I didn't get one pregenerated result), so you definitely need to figure out how to fund more prompt results.

One thing you could try is to treat this as a straightforward content farm and sell ads or sponsorship on the site. Using Google AdSense won't work since they prohibit machine generated content. You might be able to partner with a small ad network to explore specific slices of the combinatorial explosion (eg. "Pets" or "Food").

There is probably a lot you can do in terms of clever yield management such as cheaply pre-generating just the neologism or descriptive term (like, just four words) and checking to see whether it and/or the original terms reveal good search results and/or are being actively bid on through AdWords before you select which to generate the four full prompts for.

Allow users to upvote/downvote results such that disappointing ones should be shown less often or even regenerated if sufficiently unpopular. Possibly have the vote be on a specific prompt and its result rather than the entire set of four.

GPT has shown facility in determining when a prompt doesn't make sense, experimenting with that to better target your limited generation budget to "good" combinations may help:

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/

I sensed this too. So I looked at the author's profile. Now I'm even more confused:

From https://arr.am:

Stuff That’s Happened to Me

I took a photo that may now be one of the most reproduced images in history.

I once came up with an idea with a friend that accidentally became a national TV ad featuring Nicki Minaj, Serena Williams, Usher, Kylie and Kendall Jenner.

I once crashed a fashion show, pretended to be a model, and walked on-stage as dozens of photographers snapped pictures of me.

I was paid in whiskey for a live harmonica performance at a jazz bar in Tokyo. I don’t play harmonica particularly well.

I DJ’d for Chamillionaire. I don’t know how to DJ.

I was yelled at for five minutes by the founder of Skye Vodka for taking his favorite table at a restaurant.

I’ve discovered and excavated dinosaur bones in the Montana badlands.

I think there is a confidence score actually! Most blogs about it don't show them but this one went into it:

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/

It's really cool how the uncertainty prompts alter the confidence associated with the next words.

I guess I'm not disagreeing with you in the abstract that a theoretically strong enough AI could identify bad papers, especially if it had some help for 'real' arithmetic. It at least could flag the most basic issues like plagiarism, cited documents that don't contain the cited fact, etc. Detecting claims that are themselves implausible seems like the hardest task possible, however. That's very close to general AI.

This criticism was addressed by Nick Cammarata of OpenAI, who said that:

> “it’s all about the prelude before the conversation. You need to tell it what the AI is and is not capable [of]. It’s not trying to be right, it’s trying to complete what it thinks the AI would do :)”

He did some "prompt engineering" and came up with:

> ‘This is a conversation between a human and a brilliant AI. If a question is “normal” the AI answers it. If the question is “nonsense” the AI says “yo be real”’

which lead to better results. Here is an article about these "Uncertainty Prompts":

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/

This is why it’s fair to call some prompt designs bad. With a bad prompt you can’t really judge anything from the response. But better prompt design can uncover interesting behaviors. GPT-3 will generally not reject a prompt as nonsense or unreasonable, and will instead do a straight-faced continuation, probably because it has been trained on plenty of jokes, satires, dream sequences, etc. But if you specifically give it space in the prompt to reject nonsense, it’s actually quite good at rejecting nonsense.

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts