François Chollet, creator of Keras, writes essays on AI, how large language models work, and education.
HN commenters point to his LLM prompt engineering and 'LLMs as program databases' posts as a useful mental model, though one finds the framing reductive.
I would read Francois Chollet's explanation of this. It's very good: https://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering
For words that are not in the model's vocabulary, like 'fluftable', the model uses a subword tokenization strategy. It breaks down the word into smaller known subunits (subwords or characters) and represents each subunit with its own vector. By understanding the context in which 'fluftable' appears and comparing it to known words with similar subunits, the model can infer a plausible meaning for the word. This is done by analyzing the vector space in which these representations exist, observing how the vectors align or differ from those of known words.
'As always, the most important principle for understanding LLMs is that you should resist the temptation of anthropomorphizing them.'
The original article [0] is from 2023 and was already posted on HN multiple times.
Anyway, while there's something interesting in this mental model, I find this attitude below and particularly the word "rather" to be overly reductive, seeing how you could argue that intelligence in general consists of searching in an abstract database of Platonic ideas.
> Viewing LLMs as vector program databases rather than intelligent entities provides a powerful mental model for systematic improvement.
[0] https://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering
Actually it's way easier to search for the information in an LLM compared to an 8 tb harddisk.
The search is vague, and the result is nondeterministic, but in a lot of cases it's still the better method.
https://fchollet.substack.com/i/137628402/llms-as-program-databases I like the way this article puts it. "LLM is a continuous, interpolative kind of database"
This seems to have similar ideas to an post from a year ago by François Chollet: https://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering