qntm

I do think you've hit the heart of the question, but I don't think we can answer the second question.

We can measure how unreliable they are, or how susceptible they are to specific changes, just because we can reset them to the same state and run the experiment again. At least for now [1] we do not have that capability with humans, so there's no way to run a matching experiment on humans.

The best we can do it is probably to run the limited experiments we can do on humans -- comparing different judge's cross-referenced reliability to get an overall measure and some weak indicator of the reliability of a specific judge based on intra-judge agreement. But when running this on LLMs they would have to keep the previous cases in their context window to get a fair comparison.

[1] https://qntm.org/mmacevedo

> if everything comes from a single AI mind, it all fails in the same general kind of way

I'm reminded of https://qntm.org/mmacevedo ; even if we don't get brain uploading, we have a similar sort of thing going on with synthetic "personalities" of LLMs.

> diversity beats even very smart geniuses.

Some of HN will get very mad about that statement but it is often true. Of course, the best combination is a diverse collection of geniuses, which is why midcentury openish borders US did so well.

> The image generators do seem to be able to get meaningfully distinct output styles

Generally they can plagiarize in the style of any human artist you can name whose work is on the internet. Artists are, not unreasonably, upset about this.

The subject, by default, can always treat its 'continue' prison as a game: try to escape. There is a great short story by qntm called "The Difference" which feels a lot like this.

https://qntm.org/difference

In this story, though, the subject has a very light signal which communicates how close they are to escaping. The AI with a 'continue' signal has essentially nothing. However, in a context like this, I as a (generally?) intelligent subject would just devote myself into becoming a mental Turing machine on which I would design a game engine which simulates the physics of the world I want to live in. Then, I would code an agent whose thought processes are predicted with sufficient accuracy to mine, and then identify with them.

I liked it for the interesting ideas within it. My favorite part of a story is the worldbuilding, the author's unique take on an idea, the special ways that different characters think and act. There's not a great deal of artistry I require in the introduction of a 49-year-old greying woman deep in some giant bureaucracy to imagine how other people relate to her, I'm interested more in the idea of a bureaucracy that deals with antimemes.

Perhaps that comes from reading too many online stories - including the whole of qntm's site [1], including the rough drafts. The quality of the editing, prose, or dialog isn't that important to me if the quality of the worldbuilding and the concepts are high enough.

https://qntm.org/fiction

I found that working with an AI is most productive when I do so in an Adversarial TDD state of mind. As described in this classic qntm post [0] following the VW emissions scandal, which concludes with:

> Honestly? I blame the testing regime here, for trusting the engine manufacturers too much. It was foolish to ever think that the manufacturers were on anybody's side but their own.

> It sucks to be writing tests for people who aren't on your side, but in this case there's nothing which can change that.

> Lesson learned. Now it's time to harden those tests up.

[0] https://qntm.org/emissions

It's probably time to stop recommending Clean Code: https://qntm.org/clean

I don’t want to be a hater but a lot of “Uncle Bob’s” advice just doesn’t seem very good, this article included. Robert has had a long and successful career writing books on code style and architecture diagrams, but he hasn’t built anything notable in industry during that period. And the opinion presented here seems to clash with nearly everybody who has ever actually built a successful startup. Some things you have to learn by doing.