Berkeley's student machine learning group writes explainers and commentary on topics like DALL-E 2, CLIP art, and ML basics.
HN links it for explaining image-generation models; commenters cite its DALL-E 2 post on how transformers work and the CLIP art piece.
I'm with you there but we still don't know how it works, just that it does. The method though is you take a bunch of images, you plug them into a multi dimensional array (a nice way of saying a tensor), have some kind of tagging system, and when you ask the system for an answer, it will put one out for you. So for example in the astronaut riding the horse, there is, on some level, a picture of a horse with those similar pixels, that exists in the data of some object tagged 'horse.' Likewise with astronaut. What is important is that the data sets are absolutely massive, with billions of parameters.
Here's a more of a 'not 15 year old' explanation: https://ml.berkeley.edu/blog/posts/dalle2/
In my opinion, the real problem in that case was not the overfitting, but that they extrapolated from that data. They didn't have anything above Magnitude 8. (https://ml.berkeley.edu/blog/assets/tutorials/4/earthquake-fit-analysis.png)
You should never, ever extrapolate. It doesn't matter what your model is, it won't work.
On a side note, it could be that there is a breakpoint at Magnitude 7.25, where the slope of the line really changes, and a segmented linear regression is appropriate (https://en.wikipedia.org/wiki/Segmented_regression). But we would need more data to be sure, anyway.
Absolutely. These transformer models are trained to autoregressively predict the next token (text, or now audio), and so can be viewed as powerful compression schemes by combining with something like Huffman coding (see https://ml.berkeley.edu/blog/posts/dalle2/ for a lengthy but great read on how language models can be viewed as sophisticated lossless compression algorithms). The log-likelihood loss function used to train these models can quite directly be interpreted as the average bit-per-token needed to compress the dataset.
"Compression is comprehension" - Chaitin
I had a very different reaction. I had the sense this was an art world person (student or otherwise) who wasn’t technical enough to make it interesting. Many computer generated art projects are much more dynamic and produce much more passable art (https://ml.berkeley.edu/blog/posts/clip-art/). That said, I think I would have to agree with other comments here, I don’t think any deep conclusions could be drawn from these results. For starters the artworks are really repetitive and i would guess it biases respondents quickly toward the negative.
https://ml.berkeley.edu/blog/posts/dalle2/
it's "simple" because how it works is "just" brute-fucking-force. of course coming up with the architecture and making it fast (so it scales up well) is the challenge.
and scaling works .. because .. well, no one knows why (but likely because it's just a nice architecture for learning, evolution also converged on it without knowing why)
see also: https://www.gwern.net/Scaling-hypothesis
Here are parts 1, 2 & 3:
Introduction, Regression/Classification, Cost Functions, and Gradient Descent:
https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/
Perceptrons, Logistic Regression, and SVMs:
https://ml.berkeley.edu/blog/2016/12/24/tutorial-2/
Neural networks & Backpropagation:
https://ml.berkeley.edu/blog/2017/02/04/tutorial-3/