Stats for nerds
1recommend
23mentions
4people
2011–2022years
17% recommend83% neutral
Mentions per year
What Hacker News says
- Most of the best math is not trivial and, thus, usually takes some effort to understand.
There are some good authors of math for, say, calculus, linear algebra, differential equations, advanced calculus, advanced calculus mostly for applications, real analysis, optimization, probability, some topics in stochastic processes, introductory statistics, various more advanced topics in statistics.
Mostly the books are short on motivation and applications, and as a result it is too easy to spend time on material likely not worth the time unless you can be sure both to live forever and remember forever.
For calculus I liked Johnson and Kiokemeister. I taught from Protter and Morrey, and it was easier than J&K. Lots of people liked Thomas.
For linear algebra, I liked E. Nering and, then, P. Halmos, Finite Dimensional Vector Spaces which really is baby Hilbert space theory. Take Nering seriously -- he was a student of E. Artin at Princeton. His treatment of linear algebra is balanced and polished. For one of his editions, he has some group representation theory in the back, good, and some linear programming, really bad.
A lot of people like the MIT Strang book.
For advanced calculus to help when studying physics, especially electricity and magnetism and engineering, I very much liked
Tom M. Apostol, 'Mathematical Analysis: A Modern Approach to Advanced Calculus', Addison-Wesley, Reading, Massachusetts, 1957.
He has more recent versions, but for physics and engineering I like the 1957 version and don't like the later versions at all.
For ordinary differential equations, I liked
Earl A. Coddington, 'An Introduction to Ordinary Differential Equations', Prentice-Hall, Englewood Cliffs, NJ, 1961.
He makes variation of parameters look really nice -- then can understand the remark in the old movie The Day the Earth Stood Still. Ordinary differential equations is a huge, old field, and there is some question about how much of that deserves study now. Do notice that for systems of ordinary differential equations, get to apply some linear algebra in cute ways.
For advanced calculus for applications, there is the old MIT Hildebrand -- he knows what he is talking about, is easy enough to read, and a good place to go if need one of his topics.
In recent decades, the pure math departments wanted to teach advanced calculus as the theorems and proofs for freshman calculus. So there is Rudin, Principles of Mathematical Analysis, third edition (not the first two, maybe a later edition if there is one). So here's what is going on: He wants to develop the Riemann integral which is the one in freshman calculus. For that he wants to integrate over a closed interval on the real line, that is, some [a,b] which for real numbers a <= b is the set of a real numbers x so that a <= x <= b. Rudin will argue that [a,b] is a compact subset of the reals and show that the Riemann integral exists on all compact sets. So, the first chapters are big on compact sets. Then he talks about functions that are continuous and then ones that are uniformly continuous. With uniform continuity, the Riemann integral follows right away. Later he does some infinite sequences and series and then uses these for careful treatments of some important results, the exponential function, the number e, the sine and cosine, etc. Later he does integration of functions of several variables on manifolds for Stokes theorem and the related divergence theorem, a fully careful treatment of these theorems used in E&M. He does the Cartan exterior algebra: What is going on is that he wants to integrate a function g: M --> R where M is a manifold, that is, the range of some function f from some box, triangle, etc. to the space with the M. So, for this need the formula for change of variable for integrating with several variables, and that is a determinant of a square matrix. This integration is a multidimensional version of the line integral where direction of integration is important -- the exterior algebra is the multi-dimensional version of that. Can see that again in some treatments of general relativity in physics.
I like Rudin's third edition: Once know what the heck he is driving at and how he is getting there, say, as above, then his high precision is welcome.
For statistics, I suggest using some popular elementary book as a start. Then learn probability really well and from then on study particular topics in statistics as needed. The current directions in machine learning promise to make lots of particular topics important.
For a first book on statistics, consider
George W. Snedecor and William G. Cochran, 'Statistical Methods, Sixth Edition', ISBN 0-8138-1560-6, The Iowa State University Press, Ames, Iowa, 1971.
My wife did really well with that. So, get a good start on statistics and, then, get to learn some analysis of variance (experimental design), an underrated topic.
For a second book on statistics, consider
Alexander M. Mood, Franklin A. Graybill, and Duane C. Boas, 'Introduction to the Theory of Statistics, Third Edition', McGraw-Hill, New York, 1974.
Here, go quickly and get only the high points and don't expect the math to be very good -- in places it's pretty bad.
For regression analysis and linear multivariate statistics more generally, there are several books, Maurice M.\ Tatsuoka, Donald F.\ Morrison, William W.\ Cooley and Paul R.\ Lohnes, N.\ R.\ Draper and H.\ Smith. So, in particular, get enough to understand that regression is a perpendicular projection and, thus, get the Pythagorean theorem again.
For more on such statistics aimed at machine learning, get the Breiman CART -- Classification and Regression Trees, maybe much of the start of ML.
With that much in statistics, will have seen a lot of applied probability and may be ready for the real stuff. For that, need measure theory, e.g., the first half of Rudin, Real and Complex Analysis or Royden, Real Analysis. Then read Breiman, Probability. After that might read some of Chung, Loeve, Neveu, and maybe some more. Then return to applications including statistics with a really solid foundation in probability, random variables, the classic limit results, and much more. Then can read and/or write lots of advanced topics in statistics.
For optimization, a similar review is possible, but it's getting late.
- In the course, in lecture
"Reducing Loss: Gradient Descent"
is
"Convex problems have only one minimum; that is, only one place where the slope is exactly 0. That minimum is where the loss function converges."
The first sentence is flatly wrong: E.g., for positive integer n and the set of real numbers R, function f: R^n --> R where for all x in R^n f(x) = 0, f is convex, concave, and linear, and for all x in R^n x is a minimum and a maximum of f.
Can there be uncountably infinitely many alternative minima for the Google ML problems? Yes, e.g., just enter one of the independent variables twice.
The second sentence is nonsense.
Grotesque, outrageous incompetence!!!!
It has long been known that minimizing a convex function, even a differentiable convex function, with just gradient descent can be just horribly inefficient. A LOT is known about how to do much better than just gradient descent. E.g., there is Newton iteration (right, that Newton, hundreds of years ago) and quasi-Newton. And there's more.
Why so inefficient? Well, draw a picture like Google did except use just two independent variables instead of just the one in the Google picture. Then see that the resulting, convex "bowl" can be like a long, narrow boat with a very gentle slope in one direction and a very steep slope in an orthogonal direction. Yes the cross section of the bowl can be, first cut, an ellipse with one short axis and one long one. Sure, the axes are eigenvectors, etc. and the ellipse is part of a local quadratic approximation. Well, gradient descent keeps going back and forth nearly parallel to the short axis of the ellipse and making nearly no progress on the long axis. People have known this and known good things to do about it for, uh, at least half a century.
For the Google ML problems, might (1) tweak Newton iteration to improve the rate of convergence to a minimum or (2) at each iteration don't get a gradient descent but get a supporting hyperplane of the epigraph of the convex function, as the iterations proceed, accumulate these hyperplanes, notice that they lead to an approximation of the full epigraph, and use linear programming or some tweak of that to minimize the hyperplane approximation to the convex function -- there is much more that can be said here. By the way, the convex function for that ML problem is quite special, e.g., quadratic.
A lot has long been known and well polished in regression and classification, e.g., with TeX markup:
N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \
Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \
C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \
A good start on convexity is
Wendell H.\ Fleming, {\it Functions of Several Variables,\/} Addison-Wesley, Reading, Massachusetts, 1965.\ \
Total, fun, dessert ice cream on convexity is Jensen's inequality; right away can use it to prove a lot of classic inequalities.
Gee, look on the upside!!!! From this sample, the claims of machine learning (ML) revolutionizing the economy are nonsense!!!! And for startups, don't much have to worry about serious competition from Google!!!!
- > Yeah sure its a projection but so what. Why and when should a L2 norm projection be useful.
We want the answer, and more generally a predictive model.
We use the data we have to define a vector subspace. We use that subspace to creep up on the answer we want. Finally to get the answer, we project onto the subspace we have settled on. We CAN do the projection even though we don't have the answer. We take that projection on to the subspace, that point in that subspace, as our approximation to the answer.
We use L^2 because that projection, with the Pythagorean theorem, is the closest in L^2.
Being close in L^2 is a good thing: L^2 is a Hilbert space, and being Cauchy convergent means being convergent in L^2, and then there is a subsequence that converges almost surely, with probability 1, exactly, etc., and in practice it's exactly. A sequence that converges in L^2 but not exactly is pathological.
Then as we use more variables, the vector subspace gets closer to the answer and we get a better projection although maybe not a better fitting model.
To me, tails of distributions have next to nothing to do with it or the use of L^2.
In particular, the basic regression math, the normal equations, the Pythagorean theorem
total sum of squares = regression sum of squares + error sum of squares
makes no assumptions at all about distributions (might want to argue that need to be in L^2, that is, for each random variable X have E[X^2] finite, or maybe at least E[|X|] finite, that is, L^1) certainly does not assume a Gaussian.
Moreover the matrix in the normal equations has variances and covariances, but these are close to just the inner product in the Hilbert space L^2 and do NOT assume a Gaussian or anything about a distribution except merely that the expectations exist which in practice is a meager assumption, essentially always the case.
Just because are working with variances and covariance does NOT mean that are assuming a Gaussian.
Also notice that nowhere do we need to assume that our data is not discrete. Discrete data is still fine.
If in addition do have an appropriate Gaussian assumption, then at the end of the usual normal equations stuff, can get an F ratio and some t-tests that usually can help evaluate the quality of the fitting and the importance of the coefficients.
And for the model, we can get confidence intervals on the predicted values -- sure, will need a Gaussian assumption here, but there might be other approaches and otherwise we can hope or look for robust estimators.
L^2's fine with me.
More can be done, but the above is the first cut reason for using subspaces and L^2 projections although this explanation is rarely taught to students.
When ordinary regression doesn't work very well, then I'm reluctant to strain with much more. I might be willing in some circumstances to use what L. Breiman (I respect Breiman, Loeve student) has in his Classification and Regression Trees.
- SVM (support vector machines) is an application of curve fitting related to discriminate analysis in classic multi-variate statistics. The connections with trees are likely much as in the Breiman, et al., Classification and Regression Trees (CART), a classic in applied statistics. Breiman was a good mathematician, probabilist, and applied statistician.
Machine Learning is nothing like learning in humans or psychology.
The course looks like it stole the word learning from psychology and stole nearly all the rest from classic statistics and looks like stolen content relabeled.
To me, that the content of the course and text, and I have to conclude that I am quite familiar with nearly all of it, uses the word learning is inappropriate hype.
I object to the hype.
Students be warned: As I wrote, essentially all of that material is from statistics and nearly all from classic statistics and has essentially nothing to do with learning in any reasonable meaning of that word.
For computer science to make what is now broad misuse of the word learning is a big step up in hype and a big step down in academic quality and responsibility.
Ah, a controversial view for HN!
- Bourbaki student M. Talagrand has some work on approximate independence. If I were trying to do something along the lines of Probabilistic Machine Learning: Advanced Topics I would look
(1) carefully at the now classic
L. Breiman, et al., Classification and Regression Trees (CART),
and
(2) at the classic Markov limiting results, e.g., as in
E. Çinlar, Introduction to Stochastic Processes,
at least to be sure are not missing something relevant and powerful,
(3) at some of the work on sufficient statistics, of course, first via the classic Halmos and Savage paper and then at the interesting more recent work in
Robert J. Serfling, Approximation Theorems of Mathematical Statistics,
and then for the most promising
(4) very carefully at Talagrand.
(1) and (2) are old but a careful look along with more recent work may yield some directions for progress.
What Serfling develops is a bit amazing.
Then don't expect the Talagrand material to be trivial.
- Off-the-cuff, i.e. without digging deeply into a set of history of statistics books:
Tied 1st place:
• Markov chain Monte Carlo (MCMC) and the Metropolis–Hastings algorithm
• Hidden Markov Models and the Viterbi algorithm for most probable sequence in linear time
• Vapnik–Chervonenkis theory of statistical learning (Vladimir Naumovich Vapnik & Alexey Chervonenkis) and SVMs
4th place:
• Edwin Jaynes: maximum entropy for constructing priors (borderline: 1957)
Honorable mentions:
• Breiman et al.'s CART (Classification and Regression Trees) algorithm (and Quinlan's C5.0 extension)
• Box–Jenkins method (autoregressive moving average (ARMA) / autoregressive integrated moving average (ARIMA) to find the best fit of a time-series model to past values of a time series)
(The beginning of the 20th century was much more fertile in comparison - Kolmogorov, Fisher, Gosset, Aitken, Cox, de Finetti, Kullback, the Pearsons, Spearman etc.)
Related
Probability
5 recommend · 11
Optimization by Vector Space Methods
7 recommend · 24
Introduction to stochastic processes
4 recommend · 21
Nonparametrics
1 recommend · 9
A Course in Probability Theory
3 recommend · 11
Applied regression analysis
2 recommend · 8
