James Betker writes about deep learning, including diffusion models, text-to-speech with Tortoise, building GPU rigs, and why datasets drive model quality.
Commenters point to him as the builder of the Tortoise TTS model and to his detailed posts on building multi-GPU machines.
The following blogpost by an OpenAI employee can lead us to compare patterns and transistors.
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/
The ultimate model, in his (author's) sense, would suss out all patterns and then patterns among those patterns and so on, so that it
delivers on compute and compression efficiency.
To achieve compute and compression efficiency, it means LLM models have to cluster all similar patterns together and deduplicate them. This also means successively levels of pattern recognition to be done i.e. patterns among patterns among patterns and so on , so as to do the deduplication across all hierarchy it is constructed. Full trees or hierarchies won't get deduplicated but relevant regions / portions of those trees will, which implies fusing together in ideas space. This means root levels will be the most abstract patterns. This representation also means appropriate cross-pollination among different fields of studies further increasing effectiveness.
This reminds me of a point which my electronics professor made on why making transistors smaller has all the benefits and only few disadvantages. Think of these patterns as transistors. The more deduplicated and closely packed they are, the more beneficial they will be. Of course, this "packing together" is happening in mathematical space.
Another thing which patterns among patterns among patterns reminds me of homotopies.
This brilliant video by PBS Infinite Series is amazing. As…
What may be relevant in this context is the observation that models ultimately converge on their dataset:
> trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point. (…)
> This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset.
> Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset.
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/
– Is scraping good enough? Can we do without curated datasets? (This would be the most significant encyclopedic effort in human history – and we are by no means ready for this.)
Edit: It may be worth remembering the Mundaneum, https://en.wikipedia.org/wiki/Mundaneum
Fair push back, but I do think the LSTM vs Transformers point kinda supports my position in the limit, not refutes. Once the compute bottleneck is removed, LSTMs scale favourably.
https://arxiv.org/pdf/2510.02228 (I believe there's similar work done on vanilla LSTMs, but I'd have to go digging)
So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorothmic advances were useful/did unlock/remove the bottleneck.
A wider point is that eventually (once compute and data are scaled enough) the algorithms are all learning the same representations: https://arxiv.org/pdf/2405.07987
And of course the canon:
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Scaling compute & data > algorithmic cleverness
James Betker in the tortoise-tts repo, which is similar, says he spent $15k for his home rig. I'm not finding right now how long it took to train the tortoise model but feel like I read him say weeks/months somewhere. Obvs all kinds of variations depending on coding efficiency and dataset size, but another datapoint
https://nonint.com/2022/05/30/my-deep-learning-rig/ https://github.com/neonbjb/tortoise-tts
I dare say RAG with vector DBs is underwhelming because embeddings are not underrated but appropriately rated, and will not give you relevant info in every case. In fact, the way LLMs retrieve info internally [0] already works along the same principle and is a large factor in their unreliability.
[0] https://nonint.com/2023/10/18/is-the-reversal-curse-a-generalization-problem/
It's going to get worse. Here's a great primer on the problem we're facing by a Google -> OpenAI engineer.
"I think that being able to automatically produce arbitrary content that is indistinguishable from human-generated content at scale is the death knell of the web as we know it."
https://nonint.com/2023/03/09/gpt-might-be-an-information-virus/