Prime Intellect, an AI lab, posts about decentralized model training, reinforcement learning, its INTELLECT models, and recursive language models.
HN links it for decentralized training across separate clusters, its INTELLECT-3 model, and its post on recursive language models and context folding.
I mean it as in, train a model across different clusters instead of a centralized cluster. It's been shown that it's possible to train 10B models this way. If more research effort was put into this, that would be great
https://www.primeintellect.ai/blog/intellect-1
Synthetic data is already widely used to do training in the programming and mathematics domains where automated verification is possible. Here is an example of an open source verified reasoning synthetic dataset https://www.primeintellect.ai/blog/synthetic-1
Runpod is worth a look for these on demand workloads https://www.runpod.io/pricing I use a lot for ffmpeg workloads.
Found this a few days ago which might be neat for finding cheaper https://www.primeintellect.ai/
No affiliation with either
Limits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.
>The idea implies that eventually crowd-sourcing an open AI is probably technically feasible
It's already technically feasible: https://www.primeintellect.ai/blog/intellect-2
This blog explains context folding ie tool calling and recursion - https://www.primeintellect.ai/blog/rlm