MosaicML's company blog on training large language models and other AI models efficiently, including its open-source MPT models.
Commenters point to it for commercially usable open models like MPT-7B and praise its training framework and optimizations; one commenter calls it a leading option.
15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 (https://www.mosaicml.com/blog/amd-mi250) but AFAIK hasn't been tested at this kind of scale. so, validating if the MI250 in a cluster can effectively train a large scale LLM will be useful. not that it really matters to any other group, you won't get these in the cloud anytime soon.
either way - say that they have about 10M A100 hours. if they use half their budget for the biggest model as FB did with Llama (rest on testing / smaller models as they scale), then they'll budget 5M A100 hours for the biggest model, or 3x llama-2's 1.7M A100 hours (2k A100 for 21 days).
However, why wouldn't Llama-3 also be 3x bigger than Llama-2? FB's H100's are coming online; they apparently have 4k H100 in a cluster (https://www.nextplatform.com/2023/09/26/meta-platforms-is-determined-to-make-ethernet-work-for-ai/), and each H100 is about 3x the speed of an A100 (https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1). So if they provided the same 'calendar time' of 21 days over a single cluster of 4096 H100, that would be about 6.2M A100 hours for a 70B model.
there is an entirely separate question of, LLM's are functions of compute _and_ data, will they also find enough high quality _and_ kosher tokens in their low resource languages?
In the data center, I think AMD is a lot more viable than most people think. MosaicML recently did a test and were able to swap MI250s with A100s basically seamlessly, within a single training run even, and ran into no issues: https://www.mosaicml.com/blog/amd-mi250
If you have an officially supported card https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#supported-gpus and are using PyTorch, then you're pretty much good to go. Also, HIPify works pretty well these days.
I think where most people have been getting into trouble is with trying to run with unsupported cards (eg, *ALL* of AMD's consumer cards), or wanting to run on Windows. This is obviously a huge fail on AMD's part since anyone who's tried to do anything with any of those consumer cards will just assume the data center cards are the same, but they're quite different. It doesn't help that I've never seen any CDNA2 card on sale/available in retail. How does AMD ever expect to get any adoption when no developers have hardware they can write code to? It's completely mental.
MosaicML (https://www.mosaicml.com/careers) | Full Time | San Francisco / Palo Alto / San Diego / New York | Onsite
Role: Software Engineer - Cloud Platform (https://boards.greenhouse.io/mosaicml/jobs/4327398004)
In this role, you will...
- Design and build our SaaS products for efficient ML training
- Ensure reliability, security, and scalability of our SaaS platform as we build out new features and onboard customers
- Deploy technologies on top of Kubernetes, build workload-aware orchestration
- Learn about the machine learning software necessary to optimize training performance across multiple scales of hardware
MosaicML is making ML training more efficient by improving efficiency of neural network training with algorithmic methods that deliver speed, boost quality and reduce cost.
We're well funded by Lux Capital, DCVC, and Playground Global to name a few. We're growing quick, and if you're interested in learning more please reach out to: shawn [at] mosaicml.com
Pretty sure it's not "real" (model) context width.
Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model).
For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ?
Maybe someone from Anthropic or MosaicMl could throw us a bone and give a bit more detail of how these are working !
https://www.mosaicml.com/blog/mpt-7b
https://arxiv.org/abs/2005.11401
MosiacML, the latest and greatest training framework to hit the scene, is taking the machine learning world by storm. The code is top-notch, and the multiple optimizations that work right out of the box with HuggingFace models are a game-changer. It's no wonder that so many businesses are hopping on the MosiacML bandwagon - this framework is the real deal.
Personally, I've been loving the MosiacML trainer for training our Bert Models. With the composer library, scaling training is a breeze - no more worrying about OOMs or streaming data from your cloud buckets. For more information, be sure to check out https://www.mosaicml.com/blog/composer-0-12-release.
In the ever-evolving field of machine learning, staying relevant is key. Believe me, I've wasted hours debugging problems with other outdated frameworks. So if you want to stay ahead of the curve and produce top-notch models, MosiacML is the way to go.
>Why isn't there a framework that does all this automatically for you?
Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times
https://www.mosaicml.com/blog/supercharge-training-composer
Also worth checking out their “training from scratch” blog posts.
Training StableDiffusion:
https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k
Training GPT-3:
https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy