Cosma Shalizi, a statistics professor at Carnegie Mellon, keeps a long-running personal site of notebooks, reviews and essays on statistics, science and society.
HN commenters praise his review of Wolfram's A New Kind of Science as a classic and cite his write-up linking transformer attention to kernel smoothing.
Crutchfield has recorded lectures on YouTube. I thought they were quite good. I'm biased toward learning statistical mechanics and nonlinear dynamics as a starting point. Search "phy 256 a(b) crutchfield" on YouTube. Crutchfield has been doing amazing work for quite some time. If you want a critical and very entertaining read, I would check out cosma shalizi's blog. He does a lot of book reviews too that may be helpful. http://bactra.org/weblog/
Other names/references include "The nature of computation" Stuart kauffman. "Nonlinear dynamics and chaos"
Too many stat mech books to possibly recommend, for the idea of emergence you'll want something including phase transitions so... Can't go wrong with 'thermodynamics and an introduction to thermostatistics". And for stat mech with lots of complex systems diversions, Sethna's book is excellent but difficult to learn from. "Entropy, complexity, and order parameters. Names in thermo/complexity: Sethna Goldenfeld Parisi HE stanley
Finally, it's hard to talk about complex systems without talking about networks, "Networks"-Newman is unambiguously the best choice. Names in networks: Mark Newman Reka Albert Avoid barabasi especially more recent stuff.
There may be better lectures/resources out there I don't know of, but I would start with crutchields lectures. Note again I am coming from heavy physics bias.
This seems to be about easier classification tasks with not too many samples, for which TF-IDF also works well (Table 3). But more generally gzip for text modeling might make sense. Quoting http://bactra.org/notebooks/nn-attention-and-transformers.html :
> Once we have a source-coding scheme, we can "invert" it to get conditional probabilities; we could even sample from it to get a generator. (We'd need a little footwork to deal with some technicalities, but not a heck of a lot.) So something I'd really love to see done, by someone with the resources, is the following experiment:
> - Code up an implementation of Lempel-Ziv without the limitations built in to (e.g.) gzip; give it as much internal memory to build its dictionary as a large language model gets to store its parameter matrix. Call this "LLZ", for "large Lempel-Ziv".
> - Feed LLZ the same corpus of texts used to fit your favorite large language model. Let it build its dictionary from that. (This needs one pass through the corpus...)
> - Build the generator from the trained LLZ.
> - Swap in this generator for the neural network in a chatbot or similar. Call this horrible thing GLLZ.
> In terms of perplexity, GLLZ will be comparable to the neural network, because Lempel-Ziv does, in fact, do universal source coding.
Maybe someone on HN will have resources for such an experiment?
Yes, actually. It's a theme that Scott is annoyingly fond of, in his fiction and non-fiction. Unsong is entirely about kabbalistic puzzles as you know. Scott and others before him were enamored with the ideas of Leo Strauss. You can read more about Strauss' ideology of hidden truths that only the brilliant and/or worthy can perceive at the link below.
Scott Alexander uses the Kolmogorov post to have plausible deniability for his discussion of "race science". There's a reason writing "HBD" was banned in the comments section and it's that it was giving the game away to too many people.
(discussion of HBD is not banned, it's simply not permitted for that string to appear - i wonder why that might be)
http://bactra.org/weblog/algae-2014-06.html#drury-on-strauss
Have you missed this post?
http://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/
While I absolutely love this illustration (and frankly everything Jay Alammar does), it is worth recognizing there is a distinction between visualizing how a transformer (or any model really works) and what the transformer is doing.
My favorite article on the latter is Cosma Shalizi's excellent post showing that all "attention" is really doing is kernel smoothing [0]. Personally having this 'click' was a bigger insight for me than walking through this post and implementing "attention is all you need".
In a very real sense transformers are just performing compression and providing a soft lookup functionality on top of an unimaginably large dataset (basically the majority of human writing). This understanding of LLMs helps to better understand their limitations as well as their, imho untapped, usefulness.
0. http://bactra.org/notebooks/nn-attention-and-transformers.html
>A large number of natural distributions follow a power-law distribution.
This is surprisingly debatable. People like to find power-law/scale-free distributions because it implies a neat generative story, but a lot of the "evidence" for power-law distributions is pretty weak. For example, you cannot just show that a log-log plot is linear--lots of other distributions can produce similar plots.
Clauset, Shalizi, and Newman have a very readable paper where they describe 1) how to properly test for a power-law distribution, and 2) use those tests to assess the validity of some claims from the literature (spoiler: not many have "good" statistical support). Here is the paper: http://arxiv.org/abs/0706.1062 Shalizi has a short blog post describing the main results: http://bactra.org/weblog/491.html
The subject here and now is Wolfram and the irritation of many comes not from the work that Wolfram has achieved on his own but the utter lack of acknowledgement | recogonition of work done by others .. a substantial amount of work by others.
His New Kind of Science both grossly overstates the importance of cellular autonoma being Turing complete and utterly ignores a vast community of others who had similar ideas to Wolfram, many before Wolfram, and some whose work it has been said he lifted directly and claimed as his own.
Like if Michalangelo stood before the Mona Lisa and been like “look how amazing what I did was”.
See: (for example) A Rare Blend of Monster Raving Egomania and Utter Batshit Insanity http://bactra.org/reviews/wolfram/