Salesforce's AI research team publishes posts on its models and methods, including CodeGen, CodeRL, XGen, Moirai, and CausalAI.
HN commenters cite it for code-generation work like CodeRL and CodeGen, the GeDi discriminator approach, and its model training data choices.
My whole point is that the way in which AI progress can intersect itself can lead to compounding effects in ways technology has not.
One way to improve upon GPT-3 is almost certainly going to be adding discriminators into the mix:
"GeDi: A Powerful New Method for Controlling Language Models"
https://blog.salesforceairesearch.com/gedi/
And it may be that rather than it being a CNN discriminator such as in that case, it will be a second transformer:
"[2102.07074] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up"
https://arxiv.org/abs/2102.07074
My point is we may not need to reinvent some new model separate from transformers/GANs and the work so far as long as novel ways of intersecting trained models produces results that can in turn feed intersect with other models.
NVIDIA in particular has had some interesting focus on using AI to generate training data for AI.
"NVIDIA Omniverse Replicator Generates Synthetic Training Data for Robots | NVIDIA Technical Blog"
https://developer.nvidia.com/blog/generating-synthetic-datasets-isaac-sim-data-replicator/
I think it's very unlikely we need to wait 25 years for a new paradigm.
In fact, I'm willing to wager a bet that whatever new paradigm will arrive will itself be the output of AI simulating different models.
Interesting. Just a few weeks back, I was reading about their previous work https://atscaleconference.com/the-evolution-of-aiops-at-meta-beyond-the-buzz/ -- didn't realise there's more work!
Also, some more researches in the similar space by other enterprises:
Microsoft: https://yinfangchen.github.io/assets/pdf/rcacopilot_paper.pdf
Salesforce: https://blog.salesforceairesearch.com/pyrca/
Personal plug: I'm building a self-service AIOps platform for engineering teams (somewhat similar to this work by Meta). If you're looking to read more about it, visit -- https://docs.drdroid.io/docs/doctor-droid-aiops-platform
> Yes, in a very very very very very limited environment like a board game.
They work in math and coding, and these two are quite general.
> CodeRL is a new framework for program synthesis through holistic integration of pretrained language models and deep reinforcement learning. By utilizing unit test feedback as part of model training and inference, and integrating with an improved CodeT5 model, CodeRL achieves state-of-the-art results on competition-level programming tasks.
https://blog.salesforceairesearch.com/coderl/
Those look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data )
DALL-E so far has no way to create consistent characters across multiple pictures. However good news if this discriminator approach https://blog.salesforceairesearch.com/gedi/ caries over to images such constraints could be imposed. This is just a handful of follow up publications away.
Anyone could just push the repos to GitHub, including GitHub employees.
Here are links to Amazon and Salesforce
https://aws.amazon.com/codewhisperer/
https://blog.salesforceairesearch.com/codegen/