The Elements of Statistical Learning
Describes important statistical ideas in machine learning, data mining, and bioinformatics. Covers a broad range, from supervised learning (prediction), to unsupervised learning, including classification trees, neural networks, and support vector machines.
Stats for nerds
144recommend
266mentions
227people
2007–2026years
62% recommend35% neutral3% critical
Mentions per year
What Hacker News says
- A while ago a lot of the discussion about overparameterization was about explaining "double descent", the observation that test error doesn't descend monotonically and actually hits a local maximum around the point where the model has just enough parameters to interpolate the data. My favorite article about double descent looks at this in terms of splines [1]. If I can try to summarize that article: when you are designing a parametrized model to fit to data, you have a choice. You can either:
1. Avoid overparameterization by design. Manually create or choose a space of functions that has limited degrees of freedom by construction.
2. Accept overparameterization and regularize.
The latter tends to be more robust, because of the bitter lesson. It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with. The best way to approach that ideal, it turns out, is really to just let the computer figure it out via regularized optimization over an overparameterized space.
Statisticians started moving in favor of overparameterization long before deep learning got off the ground. This trend dates back at least to the machine learning bible, Elements of Statistical Learning (2001).
- Then you may enjoy Elements of Statistical Learning[1] or Kevin Murphy's books[2][3][4], which this chapter is heavily indebted to; it is mostly a short gloss of the topics covered in those books.
[1]: https://hastie.su.domains/ElemStatLearn/
[2]: https://probml.github.io/pml-book/book0.html
- Understandably I'm becoming a bit dogmatic but I'll say it again, AIMA/PRML/ESL are still the best reference textbooks for foundational AI/ML and will be for a long time.
AIMA is Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig
PRML is Pattern Recognition and Machine Learning by Christopher Bishop.
ESL is Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman.
- Dropout is roughly equivalent to layer-specific L2 regularization, and it's easy to see why: asymptotically, dropping out random neurons will achieve something similar to shrinking weights towards zero proportional to their (squared) magnitude.
Trevor Hastie's Elements of Statistical Learning has a nice proof that (for linear models) L2 regularization is also semi-equivalent to dimensionality reduction, which you could use to motivate a "simplicity prior" idea in deep learning.
Yet another way of thinking about it, in the context of ReLU units, is that a layer of ReLUs forms a truncated hyper-plane basis (like splines but in higher dimensions) in feature space, and regularization induces smoothness in this N-dimensional basis by shrinking that basis towards being a flat hyper-plane
- Grab any ML book and read the chapter on random forests. If you have the maths background (which is not particularly high for random forests) which you should if you took ML courses, it’s all going to be pretty straightforward. I think someone already mentioned Hastie, The Elements of Statistical Learning, in this thread which you can download for free and would be a good start.
- I've only quickly skimmed the course on fast.ai and it seems to me that it has a more practical approach. It also says that math is not really important and high school math should do, which i do not agree with unless you only care about how to implement different techniques and hope for the best.
To understand the subject properly, I would say that you actually need a bit of a mathematical background to be able to understand the whys and hows of ML and DL. For this, I would start with Elements of Statistical Learning [1] to get all the background.
After that, probably Andew NG's Coursera Course gives you the next steps and also how to implement all those methods. Maybe in the meantime there are better courses, but I'm not sure, I haven't looked into it in a long time.
The Deep Learning book [2] is good to get you started on all the classic DL methods, but won't really cover anything new (like vene transformers which by now are not even new anymore).
After that, you can probably start looking into trying out projects by yourself, even going on arXiv to check out newer research and familiarize yourself with the academical lingo and way to describe the maths and practicalities (imo could be way better, but academics have a big ego they cannot leave to deflate).
Also, might be important to mention that this whole thing set up like this, and made properly would take 1+ years with bachelor's knowledge already, so if you just want some basic knowledge, you could skim through some of those, but this would be the path I would recommend to get proper knowledge of all the bases, and fully understand what's happening in the field.
Related
Introduction to Statistical Learning
85 recommend · 120
Information Theory, Inference & Learning Algorithms
77 recommend · 111
Deep Learning
44 recommend · 86
An Introduction To Statistical Learning With Applications In R
13 recommend · 17
Machine learning
15 recommend · 31
Pattern Recognition and Machine Learning
80 recommend · 121
