Princeton researchers write about AI hype, limits, and policy, examining claims about model performance, safety, and progress.
Commenters call it a great substack that dissects hyped AI claims, and link it for posts on slowing AI progress and inference-time scaling.
People have a hard time understanding what zero-shot, generalization, and memorization mean. Generative models are VERY hard to generate even before we begin to look at LLMs. Let me explain and hopefully we can stop this madness.
Zero Shot:
> Zero-shot learning consists in learning how to recognize new concepts by just having a description of them.[0]
Here's an example of a zero shot task. Suppose a LLM is trained only on text. Then you fine tune for image classification, but those do not include cats (of any type). Then you ask it to classify a picture of a cat. An object it has NEVER SEEN BEFORE.
The community has been pulling a fast one recently. Recent works like Imagen, DALLE, Parti, etc have been claiming a "Zero-Shot MS-COCO". These are 100% bullshit claims. You can go look at images in the COCO dataset[1] and then search them in the LION dataset[2] (CLIP retrieval). You'll see that there are similar images that have the same classes. These models may not have seen the exact same image before, but they've seen plenty of examples. This is NOT zero shot.
Generalization:
Google's developer pages[3] defines generalization as
> Generalization refers to your model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model.
The irony being that these "Zero-Shot MS-COCO" results they give are actually good tests for generalization. Assuming the datasets these were trained on were held constant (they…
Just as a followup for those interested, after reading the paper and some more critique:
The paper is mostly ok in that it points out model drift and the importance of making sure you use API versions. One good thing is that the full dataset/methodology was published on Github: https://github.com/lchen001/LLMDrift (everyone should do this!) so it was easy to replicate/validate, but there are some issues:
* Simon Boehm stripped the markdown output from the June model output and shows that it actually performs signficantly better than the March update when the Markdown is stripped - 70% correct vs 52% correct. https://twitter.com/Si_Boehm/status/1681801371656536068 - Matei Zaharia (co-author) replies that the point of the paper is that you have to watch out for formatting changes in the LLM, but Matei also announced in the parent tweet "We found big changes including some large decreases in some problem-solving tasks," so which is it? https://twitter.com/matei_zaharia/status/1681467961905926144 - I also think the authors should have been well aware that their prominent Figure 1 would be interpreted as reduced capabilities and that they shouldn't be implying that it is. I'm just going to say it's problematic and leave it at that...
* Narayanan (and colleague Sayash Kappor) published an analysis of the Primality Test https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time - basically, the March model didn't do any better than the June model. It's just that one tends to…
What this work actually shows is that a bunch of scientists are ignorant about how LLMs work, but are rushing to publish papers about them anyway. It is ridiculous for Thirunavukarasu to draw this conclusion from GPT-4's performance on a written ophthalmology exam.
From the good folks at AI Snake Oil[1]
> Memorization is a spectrum. Even if a language model hasn’t seen an exact problem on a training set, it has inevitably seen examples that are pretty close, simply because of the size of the training corpus. That means it can get away with a much shallower level of reasoning....In some real-world tasks, shallow reasoning may be sufficient, but not always. The world is constantly changing, so if a bot is asked to analyze the legal consequences of a new technology or a new judicial decision, it doesn’t have much to draw upon. In short, as Emily Bender points out, tests designed for humans lack construct validity when applied to bots.
> On top of this, professional exams, especially the bar exam, notoriously overemphasize subject-matter knowledge and underemphasize real-world skills, which are far harder to measure in a standardized, computer-administered way. In other words, not only do these exams emphasize the wrong thing, they overemphasize precisely the thing that language models are good at.
Also[2]:
> Undoubtedly, AI and LLMs will transform every facet of what we do, from research and writing to graphic design and medical diagnosis. However, its current success in…
Rebuttal: https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci
Summary: misinfo, labor impact, and safety are real dangers of LLMs. But in each case the letter invokes speculative, futuristic risks, ignoring the version of each problem that’s already harming people. It distracts from the real issues and makes it harder to address them.
The containment mindset may have worked for nuclear risk and cloning but is not a good fit for generative AI. Further locking down models only benefits the companies that the letter seeks to regulate.
Besides, a big shift in the last 6 months is that model size is not the primary driver of abilities: it’s augmentation (LangChain etc.) And GPT3-class models can now run on iPhones. The letter ignores these developments. So a moratorium is ineffective at best and counterproductive at worst.
The author says:
> Do the signatories believe that existing AI systems or their immediate successors might wipe us all out? If they do, then the industry leaders signing this statement should immediately shut down their data centres and hand everything over to national governments. The researchers should stop trying to make existing AI systems safe, and instead call for their elimination.
But a different set of people called for a halt to AI research two months ago, and his response then (https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci) was that these concerns are "fever dreams" and we should disregard them in favor of "serious policy debates".
It seems like there's a self-reinforcing system here, where any evidence or argument in favor of existential risks from AI can be dismissed out of hand.
Having watched the discussion here for the past couple of days, I agree with frozenwind's comment. I wrote my own reflection here (the last comment):
https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci/comments
Regarding impacts upon academia, I wrote two comments here:
https://scholarlykitchen.sspnet.org/2023/04/12/guest-post-gpt-3-wrote-an-entire-paper-on-itself-should-publishers-be-concerned/
I'd also like to emphasize that it is not only about open source LLMs. I haven't followed the situation myself, but some sources say that LLMs are being developed also in the dark web, and who knows what kind of content the criminals there possess? (Borderline cases like NoiseGPT are already done openly.) Given the current geopolitical situation, the same goes for nation states. Fun times ahead indeed.