Distill

I pretty much agree with you. I have a bachelor's degree in pure mathematics from a top 15 university -- I even published research as an undergraduate. Deep learning strikes me as an experimental science and even a subset of ordinary programming more than it does a subset of mathematics or statistics or statistical learning. I've read the entirety of the Deep Learning book by Goodfellow et. al. It's a fantastic book, but I was shocked at the lack of robust theory and the relative simplicity of the math. Even an introductory analysis textbook like Rudin is considerably harder, and Rudin is regularly tackled by undergraduates at any highly ranked university.

The only reason to fret about this in my opinion is if you're a PhD machine learning engineer who doesn't want the field to open up to non-PhDs. I think data scientists are used to being able to say, "hey, if you don't have a PhD, you really can't do or understand what I do" -- deep learning represents potentially a huge culture shock to that attitude. But even the Google Brain research team has some non-PhDs now.

I do think deep learning practitioners should learn the math. I just don't think there's actually that much math to learn. Certainly, if you read through the TensorFlow MNIST tutorial and you have no idea what cross-entropy is and you don't understand what the softmax layer is for, you need to go back to the basics. But these are concepts that anyone with any reasonable engineering degree can pick up relatively…

https://distill.pub is such a platform for articles about machine learning

https://explorabl.es is a more general hub.

> especially when you can tell that the 'teacher' is also the UI designer and developer in one and there's a very high level of quality and polish. There's nothing quite like it for learning a complex subject.

Agreed. Unfortunately I think this is the inherent limitation to the explosion of such material (Bret's seminal essay [0] was almost a decade ago) - it takes an insane amount of skill, in mostly orthogonal disciplines, as well as a lot of time and effort to make it. Some tools attempt to make them a little straightforward to program, but due to the nature of it you can't really have a one size fits all solution that isn't just a general programming language + rendering engine.

And there is no "explorables industry" to fund the production of those (unlike say, textbooks), so it ends up being mostly side projects for otherwise gainfully employed developers.

It would be amazing to have eg. attempts at building an entire K-12 curriculum around explorable explanations. It's also the kind of work that probably loses in clarity and value when done by a team/committee rather than a single person dedicated to their vision.

[0] http://worrydream.com/ExplorableExplanations/

I see that this is the author of the excellent "Quantum Country" so perhaps the case being made here is obviously biased, for a good reason.

I get a sense that the author seems to argue that most book authors are not conscious of the working memory of the reader. I don't think that is true. Many "good" non-fiction books I've read actually rely on space-repetitions throughout the text quite heavily. The author, to their credit, does note this for textbooks. Often concepts in long texts are referenced in a non-linear manner, either in expectation of ideas to come or in retrospect to draw parallels & new connections. Unfortunately, the onus here is on the reader to realize and extract the maximum juice out of this non-linear flow.

The key idea of this post is to shift this burden to the "medium" and not the reader. Books can't handle this responsibility in their current form. I certainly agree with this. Distill [1] is pioneering this from the perspective of medium-reader engagement. Unsurprisingly, Michael Nielsen, a collaborator of the author of the post, is a member of the steering committee at Distill. :-)

I love these new directions in dissemination of ideas!

[1]: https://distill.pub

We can understand how and why neural networks work.

The issue is that obtaining this understanding requires a lot of effort.

OpenAI invested in trying to understand how a simple CNN works.

You can read their results here: https://distill.pub/2020/circuits/

> In the original narrative of deep learning, each neuron builds progressively more abstract, meaningful features by composing features in the preceding layer. In recent years, there’s been some skepticism of this view, but what happens if you take it really seriously?

> InceptionV1 is a classic vision model with around 10,000 unique neurons — a large number, but still on a scale that a group effort could attack. What if you simply go through the model, neuron by neuron, trying to understand each one and the connections between them? The circuits collaboration aims to find out.

Really cool seeing this & excited to really start playing with it. 2023 is going to be funky!

It's pretty wild to see how quickly the space of Generative AI Media is coming along.

I started a newsletter on the topic, called The Art of Intelligence (GPT-3 came up with the name) with the first post going out last Friday on the topic of how far are we from AI generated videos, and simulated worlds like the Holodeck given the rapid progress of these visual A.I. Thought y'all might find it interesting:
https://artofintelligence.substack.com/p/dall-e-stable-diffusion-and-the-holodeck

This type of progress also reminds me of a really lovely publication from 2017 in Distill.pub, on the topic of these A.I. enabled creation tools - I think y'all would enjoy seeing what folks were thinking even then:
https://distill.pub/2017/aia/

I hadn't heard about Works in Progress before, but looks like incredibly well researched content. Pretty stark difference between the writing and what you see on something like sfgate.com or cnn.com.

I find it pretty awesome that the Stripe is successful enough to start branching out from core product into founder hobby areas. The cynical perspective is that this is not core to the product, but I find it kind of beautiful that the founders can do a bit of what is personally interesting to them with their company.

This reminds me of another incredible resource for deep learning articles: https://distill.pub/, which unfortunately looks like is going silent.