Snowflake Engineering Blog

By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing:

i) Open-Sourced Model Weights

ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/

iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.

iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-series-exploring-mixture-of-experts-moe-c7d6b8f14d16, https://medium.com/snowflake/snowflake-arctic-cookbook-series-building-an-efficient-training-system-for-arctic-6658b9bdfcae

v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training…

Thanks!

The Bazel posts stopped for a while after I left Google, but I'm now back at a different place where I'm working with Bazel once again. So you can expect the posts on this topic to gradually come back :) (For some context, here is one: https://medium.com/snowflake/addressing-bazel-ooms-38023b736fa2)

Let's see if my plans to return to Europe in the next few years play out...

Edit: Oh, and the few posts that look like Twitter threads (they are tagged like that) are hand-crafted and were an experiment. I first wrote the threads as blog posts, ensuring each paragraph fit in a tweet, and then copy/pasted them into Twitter. I wrote them as a blog post because I wanted to have the "unrolled" version in the canonical source for future reference, without relying on those unroll apps.

smoke? it's the same as gpt4

https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral-push-the-boundary-in-text2sql-169bfa391331#:~:text=When%20we%20combine%20the%20Snowflake%20fine%2Dtuned%20model%20with%20Mistral%20Large%20in%20a%20chained%20setup%2C%20we%20are%20able%20to%20improve%20execution%20accuracy%20by%206%20percentage%20points%20from%20~40%25%20to%20%3E46%25%20in%20our%20internal%20evaluation%20dataset%20%E2%80%94%20outperforming%20GPT%2D4%3A