Andrej Karpathy writes tutorials and essays on neural networks, deep learning, and training models, including a minimal GPT implementation.
HN commenters call it the best AI blog to follow, and praise the microgpt post as a clear way to understand backprop and LLMs.
> Diet and exercise are indeed not the path to weight loss. This is well known: most fad diets work this way in some fashion or another, and it’s well known that most fad diets fail.
I think you're misunderstanding what I'm saying. I'm not suggesting people go on keto or weight watchers. Those fad diets don't necessarily fail because they're ineffective, although they probably are--they fail because they are highly prescriptive and restrictive and it's difficult for people to actually execute the diet.
What I'm saying is that reducing total food intake for 3 days creates a lasting decrease in appetite. You can prove this to yourself by skipping breakfast for a few days: after a while, you will simply not be hungry at breakfast time.
The best post I've ever seen on weight loss comes from Andrej Karpathy, head of AI at Tesla: https://karpathy.github.io/2020/06/11/biohacking-lite/
I'm just an enthusiast. With that said, I think the author's first image image really captures the core of forward propagation and back propagation. For people who want to get more of a feel on autodiffing, I find these 2 sources amazing [1, 2]. They don't use the word autodiffing, but looking at it, I can see it's the same thing.
I once made an attempt at animating feed forward propagation [3]. If I'd animate back propagation, I'd probably make an animation on autodiffing as well as that is the most intuitive way I know on how to explain back propagation right now. It's still on my big to do list ;-)
[1] http://karpathy.github.io/neuralnets/
[2] 3Blue1Brown does in episode 4 of his Deep Learning series: https://www.youtube.com/watch?v=tIeHLnjs5U8
[3] https://melvinroest.github.io/Interactive-Neural-Networks/
> Among many, many other things, he invented the term "vibecoding".
Yeah, that's a great reason to hate him, but the person you're responding to asked why his Twitter braindroppings belong on the front page.
It should be stated, again, that Karpathy completely missed the boat on LLMs, leaving OpenAI before they developed ChatGPT, and that he convinced Tesla to pursue a visual-only, no LIDR approach to FSD that doesn't work and probably won't ever work until after LIDR-based systems have already solved FSD.
Karpathy is the AI-equivalent of Sam Altman, who, for whatever reason, only fails upward. I think many HNers like him because he reminds them of themselves. Look at this bullshit and tell me it doesn't read like something the average HNer would write: http://karpathy.github.io/2020/06/11/biohacking-lite/
Personally I recommend Stanford CSI 231n
http://cs231n.stanford.edu/
Its specifically geared towards visual recognition, but it starts with the basics of machine learning and moves on to feed forward nets and covnets and covers RNNs and attention towards the end.
The assignments are a great set of jupyter notebooks that really get your hands on the material and you can find a number of peoples complete assignments on github just by searching.
The lectures are available online as well https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv
I've done hinton's and Ngs courses and as someone who already has a non-ai development background I found this to be the best introduction. Its really an extension of Andrej Karpathy's Neural Nets for Hackers (http://karpathy.github.io/neuralnets/)
Right, I admit that I don't know the first thing about ML, so I tried an experiment.
Consider a language with the tokens "{[()]}" and the following grammar:
S := S S | '{' S '}' | '[' S ']' | '(' S ')' | <empty>
That is, "[()]" and "[]()" are valid sequences, but "[(])" or "))))" aren't. A child would quickly figure out the grammar if presented some valid sequences.
I generated all 73206 valid sequences with 10 tokens and used it as input to the RNN text generator code at http://karpathy.github.io/2015/05/21/rnn-effectiveness/. After 500,000 iterations I'm still getting invalid sequences.
Am I doing something stupid, or is a RNN text generator weaker than a child (or a pushdown automaton)? Is GPT fundamentally more powerful than this?
Yes happy to share once the evaluation period is over, but sadly I dont think Im allowed to share beforehand. Will have to check when but I expect it should be in approximately 2 weeks.
In the meantime, you might want to check out this excellent blog post http://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html. This will provide you with skeleton code to implement a character level generative model (similar to Practical 3, Task 2 with the exception that there you will generate words and not characters). Andrej Karpathy's blog post on LSTMs is also excellent and I believe he also provides the code in his repository as well http://karpathy.github.io/2015/05/21/rnn-effectiveness/