A user's guide to measure theoretic probability

What Hacker News says

1. nppOct 2010
   Advanced Linear Algebra, Roman; Linear Algebra, Hoffman & Kunze; Matrix Analysis, Horn & Johnson; Principles of Mathematical Analysis, Rudin; Real Analysis, Royden.

   Miscellaneous comments:

   - Reading pure abstract algebra (e.g. Dummit & Foote) isn't a good use of time if you intend to go into statistics, since it only shows up in a few very special subareas. If you decide to go into one of these areas, you can learn this later.

   - More advanced books on linear algebra usually emphasize the abstract study of vector spaces and linear transformations. This is fine, but you also need to learn about matrix algebra (some of which is in that Horn & Johnson book) and basic matrix calculus, since in statistics, you'll frequently be manipulating matrix equations. The vector space stuff generally does not help with this, and this material isn't in standard linear algebra books. (Similarly, you should learn the basics of numerical linear algebra and optimization -- convex optimization in particular shows up a lot in statistics.)

   - People have different opinions on books like Rudin, but you need to learn to read material like this if you're going into an area like probability. It's also more or less a de facto standard, so it is worth reading partly for that reason as well. So read Rudin/Royden (or equivalent, there are a small handful of others), but supplement them with other books if you need (e.g. 'The Way of Analysis' is the complete opposite of Rudin in writing style). It helps to read a few different books on the same topic simultaneously, anyway.

   - Two books on measure-theoretic probability theory that are more readable than many of the usual suspects are "Probability with Martingales" by Williams and "A User's Guide to Measure-Theoretic Probability" by Pollard. There is also a nice book called "Probability through Problems" that develops the theory through a series of exercises.

   Show morethread
2. justokayJan 2010
   Mike Jordan at Berkeley sent me his list on what people should learn for ML. The list is definitely on the more rigorous side (ie aimed at more researchers than practitioners), but going through these books (along with the requisite programming experience) is a useful, if not painful, exercise.

   I personally think that everyone in machine learning should be (completely) familiar with essentially all of the material in the following intermediate-level statistics book:

   1.) Casella, G. and Berger, R.L. (2001). "Statistical Inference" Duxbury Press.

   For a slightly more advanced book that's quite clear on mathematical techniques, the following book is quite good:

   2.) Ferguson, T. (1996). "A Course in Large Sample Theory" Chapman & Hall/CRC.

   You'll need to learn something about asymptotics at some point, and a good starting place is:

   3.) Lehmann, E. (2004). "Elements of Large-Sample Theory" Springer.

   Those are all frequentist books. You should also read something Bayesian:

   4.) Gelman, A. et al. (2003). "Bayesian Data Analysis" Chapman & Hall/CRC.

   and you should start to read about Bayesian computation:

   5.) Robert, C. and Casella, G. (2005). "Monte Carlo Statistical Methods" Springer.

   On the probability front, a good intermediate text is:

   6.) Grimmett, G. and Stirzaker, D. (2001). "Probability and Random Processes" Oxford.

   At a more advanced level, a very good text is the following:

   7.) Pollard, D. (2001). "A User's Guide to Measure Theoretic Probability" Cambridge.

   The standard advanced textbook is Durrett, R. (2005). "Probability: Theory and Examples" Duxbury.

   Machine learning research also reposes on optimization theory. A good starting book on linear optimization that will prepare you for convex optimization:

   8.) Bertsimas, D. and Tsitsiklis, J. (1997). "Introduction to Linear Optimization" Athena.

   And then you can graduate to:

   9.) Boyd, S. and Vandenberghe, L. (2004). "Convex Optimization" Cambridge.

   Getting a full understanding of algorithmic linear algebra is also important. At some point you should feel familiar with most of the material in

   10.) Golub, G., and Van Loan, C. (1996). "Matrix Computations" Johns Hopkins.

   It's good to know some information theory. The classic is:

   11.) Cover, T. and Thomas, J. "Elements of Information Theory" Wiley.

   Finally, if you want to start to learn some more abstract math, you might want to start to learn some functional analysis (if you haven't already). Functional analysis is essentially linear algebra in infinite dimensions, and it's necessary for kernel methods, for nonparametric Bayesian methods, and for various other topics. Here's a book that I find very readable:

   12.) Kreyszig, E. (1989). "Introductory Functional Analysis with Applications" Wiley.

   Show morethread