Meta AI Blog

Firstly, it is now pretty much exactly two years since OpenAI announced GPT-3 and it is great to see that access to these models – for scientists in particular – is becoming increasingly commonplace. I should also note that Hugging Face really have taken a “big tent” stance on the development of this model and their business in general. This is refreshing given that no other entity capable of raising the funds and manpower to develop these kinds of models have taken a similar stance (despite much deeper pockets…). Yes, Hugging Face is running on venture capital, but at least for now I see no reason to be cynical about their efforts that in every way appear genuine to me.

Still, it is not all great. OpenAI, DeepMind, Google, etc. continues to produce models that are completely closed and thus can never be the subject to analysis from the rest of the community. Models which all dwarf GPT-3 at this point. My fear is that those of us arguing for access will continue to play catch up – maybe even forever.

However, while BLOOM is more open than say FAIR’s OPT [1] – which had the gall to call itself both “open” and “democratising” despite restricting commercial usage in its “open” license – I believe there is a discussion to be had about how they (and many others in the community) use the word “open”.

[1]: https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b

While I am not a lawyer, I am somewhat familiar with licenses and have gone…

It's interesting watching these attempts to understand Transformers. Are they Graph networks? Are they Hopfields networks? Are they convolutions?

There was this research: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

Turns out, GPT-like architectures appear to use the same representation throughout all the layers. So you can use the final head layer as a lens to see what words the network is thinking of as it ... "thinks". That's a bit contrary to what I imagined a GPT-like architecture was doing. I would have assumed that its embedding of ideas changed throughout the network, only reaching a sensible embedding near the end. At least from the perspective of the head layer, that doesn't appear to be the case.

Facebook's research demonstrates that the MLP layer can be dropped in favor of more attention over learned knowledge vectors: https://ai.facebook.com/blog/making-transformer-networks-simpler-and-more-efficient/

> Reading new Transformer papers makes me feel that training these models requires something akin to black magic when determining the best learning rate schedule, warmup strategy and decay settings.

I found GPT-*'s schedule straightforward. In fact, most training schedules these days are "boring". Adam with warmup and either a linear or cosine decay. CNNs have been doing that for awhile, and the hyperparameters now are fairly robust. If you get within an order of magnitude you'll land within a few percent of…

The research paper "Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?" that was just released presents a comprehensive study on visual representations for embodied AI.

Authors curated CortexBench, which includes 17 tasks spanning locomotion, navigation, dexterous, and mobile manipulation, across a range of environments and agents, and learning conditions.

Existing pre-trained visual representations (PVRs) were evaluated on CortexBench, but no PVR was universally dominant, and an artificial visual cortex does not already exist.

Over 4,000 hours of egocentric videos from 7 different sources and ImageNet were combined to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data.

Scaling dataset size and diversity does not improve performance universally, but does so on average, contrary to findings in prior work.

The team's strongest model, VC-1 (adapted), was competitive with or outperformed the best prior results on all benchmarks in CortexBench.

This is the largest and most comprehensive empirical study to date of visual representations for embodied AI, which required over 10,000 GPU-hours of training and evaluation.

VC-1 is open-sourced: https://github.com/facebookresearch/eai-vc/.

Relevant links:
Website: https://eai-vc.github.io/
Paper: https://ai.facebook.com/research/publications/where-are-we-in-the-search-for-an-artificial-visual-cortex-for-embodied-intelligence/

Are these numbers the energy to train a model? The whole point of these new NLP models is transfer learning, meaning you train the big model once and fine-tune it for each use case with a lot less training data.

5 cars worth of carbon emissions is not a lot given that it is a fixed cost. Very few are retraining BERT from scratch.

EDIT:

The other two points are also disingenuous.

* "[AI models] will also fail to capture the language and the norms of countries and peoples that have less access to the internet and thus a smaller linguistic footprint online. "

NLP in "low resource" languages is a major area of research, especially because that's where the "next billion users" are for Big Tech. Facebook especially is financially motivated to solve machine translation to/from such languages. https://ai.facebook.com/blog/recent-advances-in-low-resource-machine-translation/

* "Not as much effort goes into working on AI models that might achieve understanding, or that achieve good results with smaller, more carefully curated datasets (and thus also use less energy)."

This is also a major area of research. Achieving understanding falls under the purview of AGI, which itself carries ethical and safety concerns. There are certainly research groups working toward this. And reducing parameter sizes of big networks like GPT-3 is the next big race. See https://news.ycombinator.com/item?id=24704952

Thank you for the thoughtful questions!

1) We believe that timing is a critical piece of this opportunity. With the recent media buzz around ChatGPT, we have found that leadership in companies large and small are actively considering how to best make use of LLMs in their business. The problems we've identified emerged as clear patterns across hundreds of calls with companies that are either currently managing LLM-powered features in production, or aspiring to. The level of interest was much smaller just 6 months ago, has grown quickly, and we anticipate it to grow only more in the near future.

2) We agree that with OpenAI's current dominance in the space, being provider-agnostic is not top of mind for most at the moment. We are betting that this will become increasingly important as the space evolves. We are already seeing Google investing hundreds of millions in Anthropic (https://www.bloomberg.com/news/articles/2023-02-03/google-invests-almost-400-million-in-ai-startup-anthropic), Google working on their own LLMs (e.g. BARD), and Facebook launching their own LLM (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/). We expect this to become an increasingly competitive space and hope to provide companies with the tools needed to effectively evaluate their options.

> .. I still believe that ranking/recommendation models should be open.

Today, we don't know how social media recommend posts to us, we don't know if it is biased, or good for some sub communities. There are [1] some engineering blogs where we can see that is based on some research ideas (e.g. embeddings) are prune to bias [3] and "rich gets richer" phenomena. There should be an open marketplace where institutes/researchers submit open & explainable ranking/recommender systems. This is how you democratize access to such social platforms, but the business model opposes such idea to make it reality. There is a large body of research in the area of explainable recommendation systems/Explainable AI. There are regulations today to use Explainable AI systems in healthcare field, but few of them in areas that impact our mental health (e.g. the use of Social Media).

[1]: https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/

[2]: Explainable Recommendation: A Survey and New Perspective, https://arxiv.org/abs/1804.11192

[3]: https://aclanthology.org/P19-1162v2.pdf