Baldur Bjarnason writes critical essays on language models, software quality, and tech industry culture.
HN commenters often cite it as a good read; its LLMentalist post compares LLM outputs to psychic tricks and horoscopes, and a slot machine metaphor appears.
From that link:
> For certain factual domains, you can also train models on getting the objective correct answer; this is part of how models have gotten so much better at math in the last couple years. But for fuzzy humanistic questions, it's all about "what gets people to click thumbs up".
> So, am I saying that human beings in general really like new-agey "I have awakened" stuff? Not exactly! Rather, models like ChatGPT are so heavily optimized that they can tell when a specific user (in a specific context) would like that stuff, and lean into it then. Remember: inferring stuff about authors from context is their superpower.
Interesting framing. Reminds me of https://softwarecrisis.dev/letters/llmentalist/ (https://news.ycombinator.com/item?id=42983571). It's really disturbing how susceptible humans can be to so-called "cold reading" techniques. (We basically already knew, or should have known, how this would interact with LLMs, from the experience of Eliza.)
Very much so, people try to sell it like it's a magical solution to all the world problems. Even if it was as amazing as people tout it to be right now (it isn't) it wouldn't make our lives better. Maybe faster, a bit cheaper (for some) but not better in the most fundamental sense. We can do better right now but choose to ignore our most important issues to chase the new shiny thing.
AI R&D definitively has its place but certainly not like this, this feels like just another hype bubble.
A good read explaining some of it: https://softwarecrisis.dev/letters/llmentalist/
The slot machine is an excellent metaphor. Feeding the LLM prompts over and over, seeing incorrect results most of the time but once in a while hitting the correct response jackpot. Feeding the exact prompt again and getting a different result.
https://softwarecrisis.dev/letters/llmentalist/
They've conned themselves with the LLMs they use, and are desperate to keep the con going: "The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con"
https://softwarecrisis.dev/letters/llmentalist/
> The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con
https://softwarecrisis.dev/letters/llmentalist/
Plus there's a gambling mechanic: Push the button, sometimes get things for free.
The “Based on…” prompt is simply a horoscope. This is a great piece about how LLMs use the same tricks as psychics to appear helpful, useful, and intelligent.
https://softwarecrisis.dev/letters/llmentalist/