Nicholas Carlini

I'm on the fence about the scoring system.

You're really being tested on (at least) 4 different things:

(1) whether you think an LLM can answer the correctly

(2) whether the answer has appeared in the training set

(3) how much non-determinism will play a role in the LLM response (i.e. 0-shot capability)

(4) how rational you're feeling that day (or, how well educated you are in statistics)

I was familiar with many of these questions in my own experience, and have seen completely different outcomes from what the quiz determined was the correct answer. I agree with others here that non-determinism can really mess things up here and is really assessing a 0-shot score, which IMO understates how LLMs are actually used (iterative enhancements, many-shot Q&A).

Finally, the scoring system ticked my ego and encouraged me to try to make up for prior errors, with disastrous effects (I was well aware that I should just go with 0.5 when uncertain):

> You answered 53.57% of questions correctly with an average log-loss of 1.233. This means you would have scored better by just leaving your guesses at 50% for every single question.

> On average there were 71.22% of people who did better at this game than you... If you had been better calibrated, you could have scored in the top 14.09% [1]

The site implicitly acknowledges it's a questionable scoring mechanism when it points out:

> there are 78.09% of people who performed worse than if they had never changed the default…

I just want to emphasise two things, which are both mentioned in the article, but I still want to emphasise them as they are core to what I take from the article as someone who has been a fan boy of Nicholas for years now

1. Nicholas really does know how badly machine learning models can be made to screw up. Like, he really does. [0]

2. This is how Nicholas -- an academic researcher in the field of security of machine learning -- uses LLMs to be more efficient.

I don't know whether Nicolas works on globally scaled production systems with have specific security/data/whatever controls that need to be adhered to, or whether he even touches any proprietary code. But seeing as he heavily emphasised the "i'm a researcher doing research things" in the article -- I'd take a heavy bet that he does not. And academic / research / proof-of-concept coding has different limitations/context/needs than other areas.

I think this is a really great write up, even as someone on the anti-LLM side of the argument. I really appreciate the attempt to do a "middle of the road" post which is absolutely what the conversation needs right now (pay close attention to how this was written LLM hypers).

I don't share his experience, I still value and take enjoyment from the "digging for information" process -- it is how I learn new things. Having something give me the answer doesn't help me learn, and writing new software is a learning process for me.

I did take a pause and digested the food for…

Actual audio from the “evil dot com” example can be found here: https://nicholas.carlini.com/code/audio_adversarial_examples/

It’s only hard to hear on a phone speaker at low volume on the 50dB example. All other examples have what sounds like some sort of static or background noise.

Granted, to an average consumer it may sound like just bad audio, but it’s not imperceptible and thus can be screened for.

Most likely we’ll end up in the same perpetual update cycle as other computer security - someone finds an exploit, that exploit is either reported for a bounty or discovered in active use in the wild, a fix is implemented and exploit is added to the test suite.

For ML that will mean adding the examples of the exploits into the learning sets/providing negative re-enforcement feedbacks.

The question will be whether we can get enough of these caught before some machinery injures someone because it saw something that wasn’t there or accepted a malicious command, and it becomes a media frenzy.

> But no one really show how they're actually solving problems with LLMs and how the alternatives were worse for them. It's all claims that it's great with no further elaboration on the workflows.

To give an example, one person (a researcher at DeepMind) recently wrote about specific instances of his uses of LLMs, with anecdotes about alternatives to each example. [1] People on HN had different responses with similar claims with elaborations on how it has changed some of their workflows. [2]

While it would be interesting to see randomized controlled trials on LLM usage, hearing people's anecdotes brings to mind the (often misquoted) phrase: "The plural of anecdote is data". [3] [4]

[1] https://nicholas.carlini.com/writing/2024/how-i-use-ai.html

[2] https://news.ycombinator.com/item?id=41150317

[3] http://blog.danwin.com/don-t-forget-the-plural-of-anecdote-is-data/

[4] originally misquoted as "Anecdote is the plural of data."

It sounds like you want more broad stuff, not necessarily learning how to train models. More like learning to use them and how they work.

https://news.ycombinator.com/item?id=36195527 and

Hacker's Guide to LLMs by Jeremy from Fast.ai - https://www.youtube.com/watch?v=jkrNMKz9pWU

State of GPT by Karpathy - https://www.youtube.com/watch?v=bZQun8Y4L2A

LLMs by 3b1b - https://www.youtube.com/watch?v=LPZh9BOjkQs

Visualizing transformers by 3b1b - https://www.youtube.com/watch?v=KJtZARuO3JY

How ChatGPT trained - https://www.youtube.com/watch?v=VPRSBzXzavo

AI in a nutshell - https://www.youtube.com/watch?v=2IK3DFHRFfw

How Carlini uses LLMs - https://nicholas.carlini.com/writing/2024/how-i-use-ai.html

For staying updated:

X/Twitter & Bluesky. Go and follow people that work at OpenAI, Anthropic, Google DeepMind, and xAI.

Podcasts: No Priors, Generally Intelligent, Dwarkesh Patel, Sequoia's "Training Data"

I made this point a few months ago here, but using the words attacker and defender (builder) rather than red team and blue team: https://lobste.rs/s/i2edlt/how_i_use_ai

The asymmetry is:

An attacker only has to be right ONCE, and he wins

Conversely, the defender only has to be wrong once, and he is wrong.

So the conclusion is:

Defenders/creators are using LLMs to pump out crappy code, and not testing enough, or relying on the LLM to test itself.

Some attackers might be too dismissive of LLMs, and could accelerate their work by using them to try more things

The comment was related to these stories:

How I Use AI (11 months ago) - https://news.ycombinator.com/item?id=41150317

Carlini has the fairly rare job of being an attacker: Why I Attack - https://nicholas.carlini.com/writing/2024/why-i-attack.html