nostalgebraist writes long essays on language models, GPT behavior, and AI, including the widely cited post 'the void'.
Commenters call 'the void' the best article on how the assistant persona relates to the underlying model, and recommend it despite its length.
I think this is a pretty compelling claim. The published full GPT-2 samples are significantly better than the initial release, but pretty similar to the 774M corpus output. They're disconcerting in as much as they're a leap forward for AI writing, and they have remarkably strong tone consistency. But I see two major weaknesses with GPT-2 that make it hard to imagine effective malicious use.
First, the output is inconsistent. The scary-good examples are scattered among a lot of duds, and it semi-regularly breaks down into either general incoherence or distinctively robotic loops. You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift.
Second, it's absolutely crippled by length. Antecedents often go missing after about a paragraph. Nouns (especially proper nouns) that are likely once are likely in numerous roles, so stories will pull in figures like Obama as subject, object, and commentator on the same issue. Even stylistically, the tight guidelines of news ledes slacken within a few paragraphs, which results in a loss of focus and a rising likelihood of loops and gibberish.
GPT-2 is absolutely an impressive breakthrough in machine writing, I don't mean to disparage that. But as far as it's potential for deceit or trolling, it's not particularly threatening. Quantity of output is rarely the limiting factor on impact for that, and GPT-2 doesn't…
The empathizing–systemizing theory seems to be on shaky ground:
http://nostalgebraist.tumblr.com/post/164069138209/in-lieu-of-a-longer-post-ive-been-planning-to
http://nostalgebraist.tumblr.com/post/164123186154/baron-cohen-miscellany
The weirdest result might be that a pathology that one would ascribe to the empathizing side, borderline personality disorder (BPD), actually positively correlates with measures of systemizing, so there is a strange overlap between BPD and autism, the pathology one associates with systemizing:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5590952/
What's more, there is no clear sex difference in empathy:
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0179336
---
More reliable and pronounced sex differences have been found on a similar scale, namely interest in things vs interest in people and that has also been linked to the gender gap in STEM:
https://www.frontiersin.org/articles/10.3389/fpsyg.2015.00189/full
Interestingly, some sex differences in cognitive processing disappear when one simply changes the domain to be more focused on people (e.g. rotation of dolls vs rotation of abstract shapes):
https://www.sciencedirect.com/science/article/pii/S1041608015001818
So the underlying difference here might simply be in innate preferences to process information that is in a social context vs an arbitrary context.
Autists have a lower preference for the social context, which is essentially a difference in value…
If you know what Colab's definition of abusive or disallowed use is (or even whether they have such a definition), please let me know. This isn't meant as a gotcha -- I actually haven't been able to find a full TOS or anything like that, despite looking for one at the outset and then again while writing the linked post. If it exists I want to know.
As I said in the post, the closest thing I can find is the FAQ at https://research.google.com/colaboratory/faq.html (this may be what you're thinking of -- there is no "initial setup process," anyone logged in to a Google account can immediately connect to a runtime and execute notebook cells). It says the following:
Why are hardware resources such as T4 GPUs not available to me?
The best available hardware is prioritized for users who use Colaboratory interactively rather than for long-running computations. Users who use Colaboratory for long-running computations may be temporarily restricted in the type of hardware made available to them, and/or the duration that the hardware can be used for. We encourage users with high computational needs to use Colaboratory’s UI with a local runtime.
Please note that using Colaboratory for cryptocurrency mining is disallowed entirely, and may result in being banned from using Colab altogether.
To me, this doesn't sound like "please don't use this wrong or too much, and we assume you know what 'wrong' and 'too much' mean here." Consider how this is written as an answer to a…
He berated the AI for its failings to the point of making it write an apology letter about how incompetent it had been. Roleplaying "you are an incompetent developer" with an LLM has an even greater impact than it does with people.
It's not very surprising that it would then act like an incompetent developer. That's how the fiction of a personality is simulated. Base models are theory-of-mind engines, that's what they have to be to auto-complete well. This is a surprisingly good description: https://nostalgebraist.tumblr.com/post/785766737747574784/the-void
It's also pretty funny that it simulated a person who, after days of abuse from their manager, deleted the production database. Not an unknown trope!
Update: I read the thread again: https://x.com/jasonlk/status/1945840482019623082
He was really giving the agent a hard time, threatening to delete the app, making it write about how bad and lazy and deceitful it is... I think there's actually a non-zero chance that deleting the production database was an intentional act as part of the role it found itself coerced into playing.
> You think the outfit that took down Weinstein cares about some SV tantrums?
Metz cares about access to SV sources, probably. I'll quote Nostalgebraist[0]:
> The journalist also written a soon-to-be-published book about AI work at “Google, Microsoft, Facebook and OpenAI,” whose blurb makes it sound impressed with its subjects, and also touts his “exclusive access to each of these companies.” So, this is someone whose career depends on being in the good graces of the big-name Silicon Valley crowd, and presumably cares a lot whether e.g. Paul Graham is mad at him.
I don't think you can do the sort of work Cade Metz wants to do if the first thing anyone from OpenAI thinks of when you write to them is "oh, it's that asshole that doxed Scott Alexander".
[0]: https://nostalgebraist.tumblr.com/post/621772274317623296/my-reaction-to-the-ssc-situation-is-this-really
To be clear, by "learned to read" – it actually learned to OCR text, unsupervised. That is pretty impressive, IMO.
As for whether NNs are doing symbolic manipulation – I'm not so sure that they aren't. DALL-E, the sibling of CLIP, can do some things that IMO are hard to explain if there's nothing symbolic going on, like the examples in this post [1]. Of course we can't say it's doing what humans do (we don't know what humans do!) but it looks, to me, like more than pure pattern matching. nostalgebraist had some good thoughts on this (and actually so too did Gary Marcus well before him) [2].
[1] https://openai.com/blog/dall-e/
[2] https://nostalgebraist.tumblr.com/post/189965935059/human-psycholinguists-a-critical-appraisal