Stephen Wolfram

>The subject here and now is Wolfram and the irritation of many comes not from the work that Wolfram has achieved on his own but the utter lack of acknowledgement | recogonition of work done by others .. a substantial amount of work by others.

He wrote many essays on history of mathematics and science that IMO not just give credit where credit is due, but help immortalize these achievements.

We would be also be better off if the people who do interesting things that he wrote about bragged about them - so we'd get more exposure to the interesting things.

Ignoring bragging is easy. Finding out what Ramanujan's thought process was when he was coming up with his brilliant mathematics, in the absence of his own detailed notes, is pretty damn hard.

[1] https://writings.stephenwolfram.com/category/historical-perspectives/

As opwieurposiu said, embeddings are high-dimensional vectors. Often, they're created by classic math techniques (e.g. principal component analysis), or they are extracted from a model that proved useful for something else.

For example, a neural net model accepts a massive number of input values that directly map to the input. So those initial values don't add any info. But a layer further inside the model, with fewer values and probably close to the end, is smaller and should reflect what the model's learned. Like a lot of deep learning, three values work but don't give much insight.

If I'm wrong, I hope somebody more knowledge corrects me. I got my understanding from basic into tutorials and Wolfram's essay on ChatGPT: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

I don't see that this follows. It doesn't seem materially different than knowing that U always follows Q, and that J is always followed by a vowel in "legal" English language words.

https://content.wolfram.com/uploads/sites/43/2023/02/sw021423img21.png from https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

I imagine it's technically possible to do this in a piecewise manner that doesn't "understand" the larger board. This could theoretically be done with number lines, and not a geometry (i.e. the 8x8 grid and current state of each square mentioned in the comment you replied to). It could also be done in a piecewise manner with three ternary numbers (e.g. 1,0,-1) for each 3 square sets.

I guess this is a kind of geometric representation on the order of Shannon's Theseus.

The recent post from Stephen Wolfram[1] is pretty good as an introduction, but I haven't seen any super comprehensive material that tries do dissect all the interesting behaviour we see in the really big llms. For that just reading the relevant papers themselves has been pretty fruitful for me. Some of them are actually very well written, even if you aren't used to reading scientific papers.
I can recommend the Sparks of AGI paper[2] and the toolformer paper[3].

Obviously there's much more out there, those three things are a pretty good read.

[1]: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

[2]: https://arxiv.org/abs/2303.12712

[3]: https://arxiv.org/abs/2302.04761

This is what Stephen Wolfram concludes in a recent article about ChatGPT:

> The specific engineering of ChatGPT has made it quite compelling. But ultimately (at least until it can use outside tools) ChatGPT is “merely” pulling out some “coherent thread of text” from the “statistics of conventional wisdom” that it’s accumulated. But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of thinking behind it) are somehow simpler and more “law like” in their structure than we thought.

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

Kind of, existing things only with 0.8 points of randomness injected for "creativity", I'm overly simplifying of course but it is kind of like that.

I really recommend trying to get through this if you can: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

It raises some questions for me about how far this current model can actually scale / evolve without getting a bit too far out to be useful for human use. But it can (sometimes) connect things in ways that people have likely never seen before.

Maybe if you just clicked the button all day, you'd wind up with some really impressive new games people have never seen before.