The Gradient

There's been a fair bit of research over the last year on this topic.

An easy read on a Harvard/MIT study: https://thegradient.pub/othello/

A follow-up on more technical aspects of what's going on with it: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world

Two more studies since showing linear representations of world models:

https://arxiv.org/abs/2310.02207 (modeling space and time)

https://arxiv.org/abs/2310.06824 (modeling truth vs falsehood)

It's worth keeping in mind these are all on smaller toy models compared to something like GPT-4, so there's likely more complex versions of a similar thing going on there, we just don't know to what extent as it's a black box.

Part of the problem with evaluating the models based on responses is that they are both surface statistics/correlations and deeper processing, and often the former can obscure the latter. For example, in the first few weeks of release commentators on here were pointing out GPT-4 failed at variations of the wolf, goat, and cabbage problem. And indeed, giving it a version with a vegetarian wolf and a carnivorous goat it would still go to the classic answer of taking the goat first. But if you asked it to always repeat adjectives and nouns from the original problem together and change the nouns to emojis (, , ), it got it right every single time on the first try. So it did have the capacity to reason out variations of the problem, you just needed to bust the…

These predictions often surely make most sense and are most prevalent in a particular field of NLP research, like dialogue (open-domain, non-task oriented). Believable dialogue seems to require some kind of world knowledge, certain forms of 'common sense' (e.g. arithmetic reasoning), and the ability to track belief states and be coherent over long periods.

GPT-2 was critiqued for its inability to deal with simple arithmetic questions, as well as contradicting itself over long periods [1].

GPT-3 has similar deficiencies in terms of sensible real-world knowledge and displaying coherence both with its own previous answers and with a more general real-world knowledge:

'Within the domain of discrete language tasks, we have noticed informally that GPT-3 seems to have special difficulty with “common sense physics”, despite doing well on some datasets (such as PIQA [BZB+19]) that test this domain. Specifically GPT-3 has difficulty with questions of the type “If I put cheese into the fridge, will it melt?”.' [2]) of the type that would obviously torpedo attempts at open domain dialogue over long periods.

Take a typical two paper pointing out the failings of brute force methods for dialogue. They trot out a classic example:

'Without getting into the specifics of how these systems are constructed, it is apparent that such an approach [brute force] provides a method for imitating a dialogue but not for participating in one. We can see the limitations of the approach from examples…

Hi, my name is Alexander, I one of authors of both Gradient pieces, Open STT and silero.ai.

We are planning on adding 2-3 new languages soon (English, German, maybe Spanish), so if you would like us to fully open source everything - please support us here - https://opencollective.com/open_stt

A lot of feedback here, very nice!

As with the Russian Speech community for some reason people are willing to share their feedback on third-party forums, but do not bother to write to the author directly.

If you have something to say - please email me - aveysov@gmail.com or telegram me (@snakers4 or @snakers41) if you would a faster reply.

Also it seemed to me that some of the readers missed that there were actually 2 parts of the article - https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/.

With this all addressed, to answer some common criticisms / points raised below:

> Kaldi cool, NNs not cool

Though it is true that production solutions CAN be built with Kaldi (and I have even spoken with people who built them, as well as vosk author), we purposely omitted Kaldi because we believe that it is a technological dead-end. Also it depends a lot on antiquated technology, is very difficult for new-comers etc etc

Another problem is that ... it requires a lot of specific knowledge to add languages. Whereas our approach is just plug-and-play - just add more data!

For example - our best model in production is just 300-500 lines of code in PyTorch.

With so much…

> We have multi-paragraph autocomplete that's matching existing texts more and more closely.

OK, I think I see where you're coming from. It sounds like what you're saying is:

E. LLMs only do multi-paragraph autocomplete; they are and always will be incapable of actual thinking.

F. Any approach capable of achieving AGI will be completely different in structure. Who knows if or when this alternate approach will even be developed; and if it is developed, we'll be starting from scratch, so we'll have plenty of time to worry about progress then.

With E, again, it may or may not be true. It's worth noting that this is a theoretical argument, not an empirical one; but I think it's a reasonable assumption to start with.

However, there are actually theoretical reasons to think that E may be false. The best way to predict the weather is to have an internal model which approximates weather systems; the best way to predict the outcome of a physics problem is to have an internal model which approximates the physics of the thing you're trying to predict. And the best way to predict what a human would write next is to have a model of a human mind -- including a model of what the human mind has in its model (e.g., the state of the world).

There is some empirical data to support this argument, albeit in a very simplified manner: They trained a simple LLM to predict valid moves for Othello, and then probed it and discovered an internal Othello board being simulated inside the…

Something feels a bit off about this article.

The content, the name of the website, the logo they use — all make me initially trust them as my kind of people. I want to read this without questioning who the author is. It has the look and feel of something I might align with, instinctively.

Alas, this is a dangerous thing in the world of bogus news.

The author and his editors then refer to Founder Fathers (not Founding) and “tying” police powers (instead of “tying the hands of the police” or “curtailing the powers of the police”.) People learning English as a second language are, of course, allowed to make mistakes, but written journalism about US public policy isn’t. That’s the point where one stops and does due diligence on who these people are. This is hard, because the The Gradient’s team don’t really introduce themselves very clearly.

It would be great to have just a little bit more information about what this site is before delving into this article more. The about page suggests it’s a student/college magazine:

https://thegradient.pub/about/

Somewhere between the end of the article and the bottom of the page is a mini bio on the author. They work in policy / politics, including working in DC. They also have a public bio on LinkedIn with more details.
I don’t mean to cast unfair aspersions, but am I reading a nudge piece from a lobbying think tank?

I feel guilty for even asking. How did the internet make me so skeptical?

This is wrong, or at least a simplification to the point of removing any value.

> NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words).

They are trained to maximise this, yes.

> Their weights are a model of this distribution.

That doesn't really follow, but let's leave that.

> Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies.

Here's the rub. If how you describe them is all they're doing then a sequence of never-before-seen words would have no valid response. All words would be equally likely. It would mean that a single brand new word would result in absolute gibberish following it as there's nothing to go on.

Let's try:

Input: I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now have.

Result: You now have two kjsdhlisrnj.

I would wager a solid amount that kjsdhlisrnj never appears in the input data. If it does pick another one, it doesn't matter.

So we are learning something more general than the frequencies of sequences of tokens.

I always end up pointing to this but OthelloGPT is very interesting https://thegradient.pub/othello/

While it's trained on sequences of moves, what it does is more than just "sequence a,b,c is followed by d most often"