Jonathan Belotti, a software engineer, writes on performance, serverless, AWS outages, language model history, and developer practices.
HN commenters call his LLM history a good overview that matches how language model research played out; others discuss his argument about knowledge and old versus new software.
This is quite a good overview, and parts reflect well how things played out in language model research. It's certainly true that language models and deep learning were not considered particularly promising in NLP, which frustrated me greatly at the time since I knew otherwise!
However the article misses the first two LLMs entirely.
Radford cited CoVE, ELMo, and ULMFiT as the inspirations for GPT. ULMFiT (my paper with Sebastian Ruder) was the only one which actually fine-tuned the full language model for downstream tasks. https://thundergolfer.com/blog/the-first-llm
ULMFiT also pioneered the 3-stage approach of fine-tuning the language model using a causal LM objective and then fine-tuning that with a classification objective, which much later was used in GPT 3.5 instruct, and today is used pretty much everywhere.
The other major oversight in the article is that Dai and Le (2015) is missing -- that pre-dated even ULMFiT in fine-tuning a language model for downstream tasks, but they missed the key insight that a general purpose pretrained model using a large corpus was the critical first step.
It's also missing a key piece of the puzzle regarding attention and transformers: the memory networks paper recently had its 10th birthday and there's a nice writeup of its history here: https://x.com/tesatory/status/1911150652556026328?s=46
It came out about the same time as the Neural Turing Machines paper (https://arxiv.org/abs/1410.5401), covering similar territory -- both…
> Yeah, let's remember everything that was great about 25 years ago and forget everything that sucked. Juxtapose it with everything that sucks about today but omit everything that's great. Come on man.
That's fair. When I see bad code today, and try to explain to myself what's bad about it, I realize that people totally did the same things 25 years ago.
Nowadays there is just so much more code, and we stand on taller piles of "architecture" trying to scale higher heaps of expectations. The thing is, the effect of the bad stuff seems to compound more readily than the effect of the good stuff. And meeting the demand for more code involves broadening the base of people doing the coding.
> If you think things suck now, just make it better! The world is your playground.
I agree with this. A lot of the modern expectation is artificial, emphasizing form over function. Even where it isn't, a lot of the modern technique is unnecessary cargo-culting. You can do a lot locally if you believe in your machine (https://thundergolfer.com/computers-are-fast ; https://www.youtube.com/watch?v=ucWdfZoxsYo).
For PAPER I'm targeting Python 3.6+ (where they added `pathlib.Path` and f-strings, and upgraded the SSL version) with the intent to support it indefinitely (which involves forking certain dependencies).
This is the central claim of E.D Hirsch's Why Knowledge Matters[1] book on educational reform. Hirsch is perhaps best known for coining the term "cultural literacy" in his book of the same name.
A little while back I wrote about cultural literacy in the software industry, following the lead of Hirsch's book.[2]
1. https://hep.gse.harvard.edu/9781612509525/why-knowledge-matters/
2. https://thundergolfer.com/software/culture/2024/01/14/computing-cultural-literacy/