A Course in Probability Theory

What Hacker News says

1. graycatNov 2018
   Evar D. Nering, Linear Algebra and Matrix Linear Algebra and Matrix Theory*

   Kenneth Hoffmann And Ray Kunze. Linear Algebra, 2nd Edition, Prentice-Hall, Englewood Cliffs, New Jersey, 1971.

   https://www.zuj.edu.jo/download/linear-algebra-2nd-edition-k...

   Halmos, Finite Dimensional Vector Spaces

   George E. Forsythe and Cleve B. Moler, Computer Solution of Linear Algebraic Systems

   Paul R. Halmos, Naive Set Theory, Van Nostrand, Princeton, NJ, 1960.

   More has been done since this book, but this book is a gorgeous introduction to axiomatic set theory. So, even people who want to dig into the latest work would do well to have this as the first book. And for people wanting to read any of the more advanced material here, knowledge of this book will be from good to have to important.

   https://piazza-resources.s3.amazonaws.com/is25bi6c6o1oh/isv1...

   Walter Rudin, Principles of Mathematical Analysis

   The third edition is a lot better than the first two.

   H. L. Royden, Real Analysis: Second Edition

   Beautifully written, elegant, but maybe don't work way too hard on the exercises about upper/lower semi-continuity, and there is a better summary than Littlewood's three principles.

   Bernard R. Gelbaum and John M. H. Olmsted, Counterexamples in Analysis

   John C. Oxtoby, Measure and Category: A Survey of the Analogies between Topological and Measure Spaces

   Walter Rudin, Real and Complex Analysis

   Walter Rudin, Functional Analysis

   Leo Breiman, Probability

   Kai Lai Chung, A Course in Probability Theory, Second Edition

   Jacques Neveu, Mathematical Foundations of the Calculus of Probability

   Erhan Cinlar, Introduction to Stochastic Processes

   J. L. Doob, Stochastic Processes

   I. I. Gihman and A. V. Skorohod, The Theory of Stochastic Processes I, II

   Donald E. Knuth, The TeX book

   Donald E. Knuth, The Art of Computer Programming, Second Edition

   Leo Breiman, "Statistical Modeling: The Two Cultures," Statistical Science, Vol. 16, No. 3, 199–231, 2001.

   Paul R. Halmos, "The Theory of Unbiased Estimation", Annals of Mathematical Statistics, Volume 17, Number 1, pages 34-43, 1946.

   Paul R. Halmos and L. J. Savage, "Application of the Radon-Nikodym Theorem to the Theory of Sufficient Statistics", The Annals of Mathematical Statistics, Volume 20, Number 2 (1949), 225-241.

   https://projecteuclid.org/download/pdf_1/euclid.aoms/1177730...

   Show morethread
2. graycatApr 2017
   In their "Probability review" at

   http://ermongroup.github.io/cs228-notes/preliminaries/probab...

   I see two problems:

   (1) First Problem -- Sample Space

   Their definition of a sample space is

   "The set of all the outcomes of a random experiment. Here, each outcome ω can be thought of as a complete description of the state of the real world at the end of the experiment."

   The "complete description" part is not needed and even if included has meaning that is not clear.

   Instead, each possible experiment is one trial and one element in the set of all trials Ω. That's it: Ω is just a set of trials, and each trial is just an element of that set. There is nothing there about the outcomes of the trials.

   Next the text has

   "The sample space is Ω = {1, 2, 3, 4, 5, 6}."

   That won't work: Too soon will find that need an uncountably infinite sample space. Indeed an early exercise is that the set of all events cannot be countably infinite.

   Indeed, a big question was, can there be a sample space big enough to discuss random variables as desired? The answer is yes and is given in the famous Kolomogorov extension theorem.

   (2) Second Problem -- Notation

   An event A is an element of the set of all events F and a subset of the sample space Ω.

   Then a probability measure P or just a probability is a function P: F --> [0,1] that is, the closed interval [0,1].

   So, we can write the probability of event A by P(A). Fine.

   Or, given events A and B, we can consider the event C = A U B and, thus, write P(C) = P(A U B). Fine.

   But the notes have P(1,2,3,4), and that is undefined in the notes and, really, in the rest of probability. Why? Because

   1,2,3,4,

   is not an event.

   For the set of real numbers R, a real random variable X: Ω --> R (that is measurable with respect to the sigma algebra F and a specified sigma algebra in R, usually the Borel sets, the smallest sigma algebra containing the open sets, or the Lebesgue measurable sets).

   Then an event would be X in {1,2,3,4} subset of R or the set of all ω in Ω so that X(ω) in {1,2,3,4} or

   {ω| X(ω) in {1,2,3,4} }

   or the inverse image of {1,2,3,4} under X -- could write this all more clearly if had all of D. Knuth's TeX.

   in which case we could write

   P(X in {1,2,3,4})

   When the elementary notation is bad, a bit tough to take the more advanced parts seriously.

   A polished, elegant treatment of these basics is early in

   Jacques Neveu, Mathematical Foundations of the Calculus of Probability, Holden-Day, San Francisco, 1965.

   Neveu was a student of M. Loeve at Berkeley, and can also see Loeve, Probability Theory, I and II, Springer-Verlag. A fellow student of Neveu at Berkeley under Loeve was L. Breiman, so can also see Breiman, Probability, SIAM.

   These notes are from Stanford. But there have long been people at Stanford, e.g., K. Chung, who have these basics in very clear, solid, and polished terms, e.g.,

   Kai Lai Chung, A Course in Probability Theory, Second Edition, ISBN 0-12-174650-X, Academic Press, New York, 1974.

   K. L. Chung and R. J. Williams, Introduction to Stochastic Integration, Second Edition, ISBN 0-8176-3386-3, Birkhaüser, Boston, 1990.

   Kai Lai Chung, Lectures from Markov Processes to Brownian Motion, ISBN 0-387-90618-5, Springer-Verlag, New York, 1982.

   Show morethread
3. graycatNov 2016
   product space. Two examples are the real line and the 3 space we live in (at least locally ignoring general relativity).

   But the biggie points about the Hilbert space definition are (A) the other, somewhat astounding (vector space), examples and (B) how much can do in Hilbert space that is close to good old high school plane (2D) and solid (3D) geometry and close to good, old freshman calculus with limits, convergence, etc. So, in particular, in Hilbert space we get the Pythagorean theorem of plane geometry and perpendicular projections as shortest distance from a point to a plane in solid geometry -- two biggies for pure/applied math. We get the triangle inequality. With the inner product, we get angles and orthogonality. And the core data we need for projections are just some inner products.

   The most important examples of a Hilbert space are, for positive integer n, the set of real numbers R and the set of complex numbers C, the vector spaces of linear algebra R^n and C^n -- e.g., C^n is the set of all n-tuples of real numbers. Then as in whatever first course you had that talked about vectors and dot (inner, scalar) products, with those inner products R^n and C^n are Hilbert spaces.

   The part about complete is a generalization of completeness in the real numbers, that is, the biggie way the real numbers are better than just the rational numbers. In short, for example, intuitively, in the real numbers, if a sequence appears to converge, then there really is a real number there for it to converge to. Of course, that's not true in the rational numbers since, e.g., can have a sequence of rational numbers that converges to the square root of 2 but doesn't really converge in the rational numbers because square of 2 is not a rational number. This stuff about "appears to converge" is called Cauchy convergence and is a weak definition of convergence. The point about completeness is that Cauchy convergent sequences really are convergent, that is, have something to converge to and do converge to it (essentially in the sense of limits you saw in calculus or high school algebra). If are taking limits to define or approximate what really want, then also really want completeness so that what converge to exists. So, that's completeness -- for a Hilbert space, we insist on that.

   Of course, need a background in linear algebra. So, for a first book, get any of the popular ones. If you wish, concentrate on the more geometrical parts and do less on the algebraic parts -- e.g., if there is a chapter on group representation theory, Galois theory, linear algebra over finite fields, where things go wrong when the field is the rationals, or algebraic coding theory, then feel free to leave that material for later. Likely should pay attention to dual spaces, but if wish can go light on adjoint transformations since they are less interesting when have an inner product. Curiously, can go light on change of basis for linear transformations, that is, the difference between vectors and coordinate vectors. Concentrate on dimension, linear independence, linear transformations, maybe touch on quotient spaces, use Gauss elimination as a good example of such things, eigenvalues and eigenvectors, orthogonality, and the polar decomposition (the core of factor analysis, singular value decomposition, matrix condition number, and more). If there is a little on the associated numerical analysis, then go ahead -- e.g., learn to accumulate inner products in double precision -- sure, take 10 minutes to see how to add iterative improvement to Gauss elimination and matrix inversion. For pseudo-inverses, cute material, but, especially for your question, likely won't see it again and can skip it.

   Then if have some time, take a fast pass through the classic, Halmos, Finite Dimensional Vector Spaces. It was written in 1942 when Halmos had just gotten his Ph.D. under J. Doob (e.g., as in Stochastic Processes and, more recently, Classical Potential Theory and Its Probabilistic Counterpart) and showed up at the Princeton Institute of Advanced Study and asked to be an assistant to John von Neumann, likely the inventor of Hilbert space. Well, Halmos wrote his book to be a finite dimensional version of linear algebra as if it were Hilbert space, which is commonly infinite dimensional. So you get a gentle introduction to Hilbert space. You get good at eigenvalues and eigenvectors, the polar decomposition, orthogonality, transformations that preserve distances and angles, etc. You get a lot of geometric intuition.

   BTW, at one time, Harvard's Math 55 used Halmos, Baby Rudin (below), and a book by Spivak as the three main references.

   Get a start on probability and statistics. A college junior level course should be sufficient. Don't take the course too seriously since will redo all the good parts from a much better foundation soon! Note: Elementary stat courses common get all wound up about probability distributions. Well, they do exist, are at the core of probability theory, and are very important, both in theory and conceptually, but, actually, in practice usually they require more data than you will likely have, especially in dimensions above 1. So, in practice, mostly can't actually see the actual data of the distributions you are working with! You should hear about the uniform, exponential, chi-squared, and Gaussian and go light on the rest. The Gaussian is profound and won't go away even in practice although is less important in practice than long assumed in, say, educational statistics.

   Then take a good pass through at least the first parts of Rudin, Principles of Mathematical Analysis, AKA Baby Rudin. For the exterior algebra in the back of the more recent editions, well, likely get that elsewhere, say, now in English, directly from Cartan. The first parts of Baby Rudin cover metric spaces well enough. So, in Baby Rudin, get good at working with the limits, completeness property, compactness, etc. of mathematical analysis, that is, not algebra, geometry, topology, or foundations, although the metric space material is the same as part of the part of topology called point set topology.

   Then learn more in mathematical analysis, in particular, measure theory. Measure theory essentially replaces the Riemann integral you learned in calculus and Baby Rudin. 'Bout time! Net, measure theory is a slightly different way to use limits to define the integral (areas, volumes, etc.) -- the first, biggie difference is that do the partitioning on the Y axis instead of on the X axis. The biggie reason: The resulting integral easily handles the pathological cases, especially involving limits, that the Riemann integral struggles with. Don't worry: The integral of, say, x^2 over [0,1] is still the same, IIRC, 1/3rd, right? But consider the function f: [0,1] --> R where f(x) = 1 if x is rational and 0 otherwise. Then the Riemann integral of f over [0,1] does not exist, but the measure theory integral does and gives 0 for the result.

   There are at least two now classic books, Royden, Real Analysis and Rudin, Real and Complex Analysis. But there are more, and likely more can be written. For Rudin, can f'get about the last half on functions of a complex variable.

   Royden is easier to read. Rudin has the math more succinctly presented. Some people believe that Rudin is a bit too severe for a first version; but if get used to how Rudin writes, he's really good.

   There in Rudin get good introductions to Banach space (a complete, normed linear space, that is, assume a little less than for a Hilbert space) with a few really surprising theorems, Hilbert space with an isomorphic argument that they are really all the same, a really nice chapter on Fourier theory (Baby Rudin does Fourier series; R&CA does the Fourier integral), and some nice applications.

   With that background in measure theory, then take your good pass through probability. So, probability becomes a measure as in measure theory except the values are always real and in [0.1]. A random variable finally gets a solid, mathematical definition -- it's just a measurable function (measurable is very general; in practice and even in nearly all of theory, essentially every function is measurable; in the usual contexts, it takes some cleverness to think of a function that is not measurable). And expectation is just the measure theory integral (with meager assumptions).

   And in probability get some assumptions don't see in measure theory -- independence and conditional independence, and these two yield wonders nothing like in just measure theory.

   Go somewhere; look at something; get a number; then that's the value of a random variable. In practice, suppose there are 20 random variables, you have numerical values for 19 of them, and you can argue that those 19 are not independent of the 20th and that you have some data on how they are dependent, then you have a shot at estimating the 20th. Presto, bingo, get a rich as James Simons, do machine learning, get big houses, Cadillacs, Ferraris, a yacht, a private jet -- maybe!

   Also get to use the Radon-Nikodym theorem (proved in both Royden and Rudin, and in Rudin with von Neumann's cute proof) for the grown up version of conditioning (i.e., Bayesian) and, from there, in stochastic processes, Markov processes and martingales (astounding results).

   Books include L. Breiman, Probability, K. Chung, A Course in Probability Theory, J. Neveu, Mathematical Foundations of the Calculus of Probability. And there are more. IIRC Breiman and Neveu were both students of M. Loeve at Berkeley -- sure, can also get Loeve's Probability in two volumes. Of these, Neveu is my favorite; it's elegant; but for most readers it is too succinct.

   Hilbert space again? It turns out, the set of all real valued random variables X such that E[X^2] is finite is a Hilbert space. Yes, completeness holds; with some thought, that result seems astounding, like there is no way it could be true, but it is.

   Now just derive grown up versions of most of the main results of elementary statistics for yourself from what you have learned. E.g., for the Neyman-Pearson result on most powerful hypothesis testing, just use the Hahn decomposition from the Radon-Nikodym theorem.

   And, with the Radon-Nikodym theorem, get a grown up version of sufficient statistics, right, based on a classic paper by Halmos and Savage.

   Along the way will notice that, the last time I looked, Baby Rudin defined the Riemann integral on closed intervals of finite length, but right away probability and statistics want to integrate on the whole real line, the whole plane, etc., do change of variable manipulations with such integrals, etc. Well, for the prerequisites, those are in measure theory where the first version of its integral applies also on the whole real line, the whole plane, and much more. Measure theory also give you the clean, powerful versions of differentiation under the integral sign and interchange or order of integration.

   Ah, why bother to teach the Riemann integral at all? :-)

   Show morethread
4. graycatOct 2015
   No problem. For the foundations I outlined, can work just fine with continuous functions, measurable functions, stochastic processes, random variables taking values on the real line, in the complex plane, in finite dimensional real or complex vector spaces with, say, the usual topology, Hilbert and Banach spaces, etc. Can do multi-dimensional Markov processes, and much more.

   And you can have each point on the real line an event. Fine. But you just can't take the uncountable union of any set of such events and assume that the result is also an event.

   As for the event a random variable takes a value >= 0? Fine.

   Or, let the Borel subsets of the real line be the smallest sigma algebra that contains all the open sets, e.g., all the open intervals. Then for Borel set A and real valued random variable X, can ask for the probability X is in A.

   I believe you will find that you will have a solid foundation for what you want.

   To see all this stuff, need more than just the sparse definitions and, instead, need an actual text and maybe a course. Recently looked at the on-line materials from MIT and didn't see such a course. Graduate probability is not all that popular in the US; stochastic processes in continuous time is still less popular.

   To study graduate probability, I'd recommend a good undergraduate major in pure math with good coverage of, say, W. Rudin, Principles of Mathematical Analysis. Then good coverage of linear algebra from more than one of the best known texts. Likely also spend as much time as you can in Halmos, Finite Dimensional Vector Spaces. E.g., at one time, Halmos, Rudin, and Spivak, Calculus on Manifolds were the three main texts for Harvard's famous Math 55.

   Get good with proving the theorems.

   I also recommend Fleming, Functions of Several Variables.

   Then, sure, Royden, Real Analysis. Couldn't be prettier.

   If not in a hurry, then the real half of Rudin's Real and Complex Analysis. Especially if you like Fourier theory!

   Then of the probability books, I believe that the nicest, first book is L. Breiman, Probability. He wrote that before he went consulting and came back and did CART and random forests.

   Next, K. Chung, A Course in Probability Theory. Next, J. Neveu, Mathematical Foundations of the Calculus of Probability. Then, Loeve, Probability Theory.

   Loeve is huge -- mostly just use it for reference or browse. E.g., it has sufficient statistics and stationary stochastic processes (the EEs love that) IIRC not in the other books.

   IIRC, both Breiman and Neveu were Loeve students at Berkeley.

   If do well with Breiman, then for graduate probability, likely can stop there. Else, Chung will then be fast and easy reading and reinforce what you learned in Breiman. Neveu is elegant; my favorite, but deserve extra credit for each workable exercise you can find, not actually work, you understand, just find! Sure, some of the exercises are terrific, half a course in a few lines of an exercise. E.g., he has one of those on statistical decision theory or some such. And see the Tulcea material in the back.

   Then there's more that you can do on stochastic processes, potential theory via Brownian motion, e.g., for mathematical finance, stochastic optimal control, and more.

   Show morethread
5. NY_EntrepreneurJul 2011
   "Can you please suggest on-line resource to learn great beginning and advanced probability concepts?"

   Yup! The subject at the level you ask is a major topic in applied math but is not very popular in US universities. So, you will not get many good answers. In particular, the 'computer science' community, with 'machine learning' and 'artificial' this, Bayesian that, likely won't have good answers.

   Your "on-line" part is asking a bit much; I can give you references to books but not all on-line. There may be some PDF files on-line, from TeX, that have such material; try some Google searching with the keywords used here.

   The intuitive foundations of probability went back to gambling.

   'Probability' is a field of 'applied' math and as such is well defined:

   About 100 years ago, E. Borel student H. Lebesgue invented 'measure theory' which essentially 'rewrote' classic calculus, especially the part about integration. For the simple cases, what Lebesgue did gets the same numerical values as the classic Riemann integral. The difference is that in theoretical work Lebesgue's integral is much more general and much better 'behaved'.

   But 'measure theory' has to do with, intuitively, 'length', 'area', 'volume' and various generalizations of these. Well, in probability, for probability P and event A, the 'probability' of A is P(A) and is a number in [0,1] and acts much like the 'area' of A. The connection is so close that, in the end, we have to accept that the foundations of probability are measure theory.

   Then

   A. N. Kolmogorov, 'Foundations of the Theory of Probability, Second English Edition', Chelsea Publishing Company, New York, 1956. English translation of the original German '"Grundbegriffe der Wahrscheinlichkeitrechnung,"' 'Ergebnisse Der Mathematik', 1933.

   applied Lebesgue's work to make probability a solid field of math. Since then Kolmogorov's foundations have been nearly the only one taken seriously in any 'modern' or 'advanced' work in probability, stochastic processes, or mathematical statistics.

   A good start on a good text in stochastic processes was:

   J. L. Doob, 'Stochastic Processes', John Wiley and Sons, New York, 1953.

   Doob was long at University of Illinois. One of his students was P. Halmos who was later an assistant to von Neumann at the Institute for Advanced Study and in about 1942 wrote the first version of the still standard:

   Paul R. Halmos, 'Finite-Dimensional Vector Spaces, Second Edition', D. Van Nostrand Company, Inc., Princeton, New Jersey, 1958.

   Later he wrote:

   Paul R. Halmos, 'Measure Theory', D. Van Nostrand Company, Inc., Princeton, NJ, 1950.

   with at the end a NICE introduction to probability and stochastic processes based on measure theory.

   Likely the first rock solid, quite comprehensive, highly polished presentation of 'modern' probability was the first edition of:

   M. Loeve, 'Probability Theory, I and II, 4th Edition', Springer-Verlag, New York, 1977.

   Loeve was long at Berkeley. One of his students did:

   Jacques Neveu, 'Mathematical Foundations of the Calculus of Probability', Holden-Day, San Francisco, 1965.

   and another did:

   Leo Breiman, 'Probability', ISBN 0-89871-296-3, SIAM, Philadelphia, 1992.

   Either of these two can be regarded as a more succinct presentation of the more important material in Loeve. Breiman is the more 'practical' and 'accessible'; Neveu is a crown jewel of elegance and succinctness but not always easy to read.

   Other good presentations of much the same material include:

   Kai Lai Chung, 'A Course in Probability Theory, Second Edition', ISBN 0-12-174650-X, Academic Press, New York, 1974.

   and

   Yuan Shih Chow and Henry Teicher, 'Probability Theory: Independence, Interchangeability, Martingales', ISBN 0-387-90331-3, Springer-Verlag, New York, 1978.

   In total, those texts nail down 'probability' at all four corners and make it a rock solid topic in applied math. Good knowledge of, say, Breiman is a necessary and sufficient condition for knowing 'probability' at a serious level, that is, without being watered down for 'general audiences'.

   For more, proceed with stochastic processes, stochastic optimal control, mathematical statistics, etc.

   For learning probability, minimal prerequisites (more would be helpful) would be abstract algebra, linear algebra, 'analysis', measure theory, and functional analysis.

   For abstract algebra, there are many texts. Sufficient is

   I. N. Herstein, 'Topics in Algebra', Blaisdell, New York, 1964.

   but can consider S. Lang, etc.

   For linear algebra, there are many texts, but the old:

   Paul R. Halmos, 'Finite-Dimensional Vector Spaces, Second Edition', D. Van Nostrand Company, Inc., Princeton, New Jersey, 1958.

   remains a good 'second' text. It is also an introduction to functional analysis.

   The main text for 'analysis' is just:

   Walter Rudin, 'Principles of Mathematical Analysis', McGraw-Hill, New York.

   in whatever is the latest edition.

   Can skip the material on exterior algebra or just get it from the original source, now in English:

   Henri Cartan, 'Differential Forms', ISBN 0-486-45010-4, Dover, Mineola, NY, 2006.

   For measure theory and functional analysis, the standards are:

   H. L. Royden, 'Real Analysis: Second Edition', Macmillan, New York.

   and the first (real) half of:

   Walter Rudin, 'Real and Complex Analysis', ISBN 07-054232-5, McGraw-Hill, New York.

   Actually, Loeve also covers much of this material. Neveu slips in quite a lot.

   Then you will be ready for Breiman, Neveu, Chung, Chow and Teicher, or Loeve.

   You will discover:

   The start of probability theory is a 'probability space'. For that, there are three parts, (1) the 'sample space', (2) the 'events', and (3) the 'probability measure'.

   The 'sample space' is just a set of points; to support any reasonably serious work in probability, the sample space has to be uncountably infinite. In all our work, we do some one 'trial', and that corresponds to some one point in the sample space. The other points in the sample space are what 'might' have been our trial.

   An 'event' is a set of trials. We ask that the collection of all events have some 'closure' properties to make that collection a 'sigma algebra'. So, briefly, the collection of events has as one event the whole sample space and is closed under relative complements and countable unions.

   In terms of measure theory, the sample space and the sigma algebra of events form a 'measurable space'. A 'probability measure' is just a positive measure with total 'mass' 1. Thank you Lebesgue and measure theory.

   On the reals R, we can take the open sets and, then, the Borel sets, that is, the smallest sigma algebra that contains the open sets. A 'random variable' is a function from the probability space to the reals that is 'measurable', that is, the inverse image of a Borel set is an event.

   As an alternative, we can ask that the inverse image of the Lebesgue measurable sets of R be events.

   If X is such a random variable, then its 'expectation' E[X] is just the Lebesgue ('abstract') integral of X over the probability space. For E[X] to exist, we need only that X be measurable and that the integral of both the positive and the negative parts of X not be infinite (we don't want to subtract one infinity from another). E.g., Lebesgue integration is very 'general': All we need is measurability and not subtract one infinity from another.

   By a change of variable, we can also write the expectation as a Lebesgue integral over the real line with respect to the 'distribution' of X on R. Or, X 'induces' a probability measure on R.

   We can discuss convergence of random variables -- in distribution, probability, mean square, and almost surely.

   Then we can define independence for events, sigma algebras, and random variables. E.g., with this approach, if X and Y are independent random variables and f and g are functions, then f(X) and g(Y) are independent random variables. Don't try to prove this the elementary way!

   With independent random variables, we can cover the classic limit theorems -- central limit theorem, weak and strong laws of large numbers, and the law of the iterated logarithm.

   Using the Radon-Nikodym theorem of measure theory, for random variables X and Y we can define their 'conditional expectation' E[Y|X].

   With such 'conditioning' we can discuss Markov processes, martingales, and the martingale convergence theorem.

   We can also move on to ergodic theory and, say, Poincare recurrence (keep stirring the coffee and the cream will separate back, close to where it was when it was poured in).

   Those are the previews of coming attractions.

   Note: This material is not very popular in US universities.

   Show morethread
6. HilbertSpaceNov 2009
   Early in my career, I asked that question. Eventually I got some answers.

   Broadly usually the 'big bucks' are in things that are new at least in some important sense. To find such things, one recommended approach is to stand "on the shoulders of giants". Then you may still do some original work, but you have one heck of a base. My view is that Moore's law, the Internet, etc. are not nearly fully exploited, that enormous value remains, and the best way to get some of this value is to find a big unsolved problem where a good solution will be very valuable, stand on the shoulders of some giants, do some original work to get some powerful 'secret sauce' for solving the problem (which appears not to be easy to solve otherwise), start a business, please customers, get revenue, grow the business and sell it.

   Here, however, just what problem, giants, original work, secret sauce are NOT easy to see. Broadly we are trying to see, no, MAKE, the future, and usually that is not easy. While we can see what applications have been made of statistics in the past, what statistics will be applied where in the future is not easy to see.

   Yes, broadly statistics is highly promising. Or, in 'information technology' we want valuable, new information, are awash in cycles, bytes, bandwidth, infrastructure software, and data, and want to use these to create the information. Here, on appropriate problems, statistics can commonly, easily, totally 'blow away' anything else. The marriage between (A) 'information technology' in business and (B) statistics has yet to be consummated, even to reach to 'going steady' stage, assuming the traditional order of events.

   For 100 years or so, people have come to statistics from various areas of work. Usually they had some data and some questions. Some of the areas have been educational and psychological testing, experiments and testing in agriculture and medicine, industrial quality control, model building in economics, experimental work in physics and chemistry, investing, attempts to create mathematically based sciences in the social sciences, especially economics, psychology, sociology, and political science. In sociology, old examples would be James Coleman, Pete Rossi (professors of my wife when she got her Ph.D.), Leo Goodman.

   Likely the best fast, practical path into statistics is via books, courses, etc. intended for students in the social sciences. These students commonly do not have good backgrounds in mathematics. For the mathematical prerequisites, generally can get by, as a start, with just high school first year algebra. With this path can learn about probability distributions, the central limit theorem, the law of large numbers, statistical estimation and confidence intervals, hypothesis testing, cross tabulation, analysis of variance, regression analysis, principle components analysis, and more.

   For more, statistics is a rock solid part of mathematics, as solid as any part of pure mathematics, e.g., topology, geometry, analysis, algebra, etc. That is, statistics is based solidly on theorems and proofs, sometimes relatively deep ones.

   Statistics as theorems and proofs is called 'mathematical statistics'. Long standard has been (with TeX markup):

   Alexander M.\ Mood, Franklin A.\ Graybill, and Duane C.\ Boas, {\it Introduction to the Theory of Statistics, Third Edition,\/} McGraw-Hill, New York.\ \

   The main prerequisite for this book is just a not very good course in calculus, and the book actually makes not much use of calculus. Mostly all a student will need from calculus is that it can find the area under a curve. Since the book has long been standard, can't really ignore it, but it's ugly. And often, with just calculus, the book doesn't really give solid proofs of the results. E.g., their treatment of sufficient statistics has some nice intuition, but their proof is junk. The subject cries out for a good book, but I'm not trying to write one or waiting for someone else to.

   Can get some of the flavor of mathematical statistics done with high quality, as mathematics, in, say (with TeX markup):

   Jean-Ren\'e Barra, {\it Mathematical Basis of Statistics,\/} ISBN 0-12-079240-0, Academic Press, New York.\ \

   Robert J.\ Serfling, {\it Approximation Theorems of Mathematical Statistics,\/} ISBN 0-471-02403-1, John Wiley and Sons, New York.\ \

   P.\ Billingsley, {\it Convergence of Probability Measures, 2\raise0.5ex\hbox{ed},\/} ISBN: 0-471-19745-9, John Wiley, New York.\ \

   R.\ S.\ Lipster and A.\ N.\ Shiryayev, {\it Statistics of Random Processes I, II,} ISBN 0-387-90226-0, Springer-Verlag, New York.\ \

   However, pursued mathematically, statistics has some relatively advanced prerequisites some of which curiously are not popular in US university mathematics departments.

   For the prerequisites,

   High School. Should have had high school first and second year algebra (reasonable facility with algebraic manipulations, the binomial theorem, complex numbers, both of which will see again in important ways), plane geometry (where nearly all the work was proofs -- first place to learn about proofs), trigonometry (usually assumed in calculus and important in, say, analysis of organ tone harmonics and, thus, the most important example of an infinite dimensional Hilbert space), analytic geometry (especially the conic sections, especially ellipses which definitely will see again), and, if can, solid geometry (for more intuition in three dimensions).

   College. Need a standard calculus course, not necessarily a very comprehensive or difficult one because will do the subject all over again, maybe two or three times, and more later, WITH the proofs!

   Then need linear algebra, that is, how to work with data of several dimensions, which is just crucial. The big result is the polar decomposition, and there get to think about ellipses and get to use complex numbers. Also the course is an introduction to functional analysis and Hilbert space. Use any popular book to get started but in the end cover the classic, Halmos, "Finite Dimensional Vector Spaces'. Halmos wrote this when he was an assistant to von Neumann and intended it to be a finite dimensional introduction to Hilbert space (which once von Neumann had to explain to Hilbert) which it is. It also has some multi-linear algebra, of interest to exterior algebra now popular in relativity, but likely for nearly any business applications of statistics for the next several decades can skip that chapter.

   Then need some advanced calculus. That is a poorly organized, huge, catch-all subject beyond any one course. The usual start is 'baby' Rudin, 'Principles of Mathematical Analysis'. So, that's calculus with the proofs. Warning: The book is severe, succinct, with zero pictures. Have to draw your own pictures in your head. The book is packed solidly with powerful material just awash in important applications from statistics, economics, and engineering to physics, but there is hardly a hint of the applications in the book. I enjoyed the book, but few people will enjoy it or even get through it. Hint: Get a really good teacher! Then for more, popular is Spivak, 'Calculus on Manifolds', mostly because it is short. Actually, it's too short. I prefer Fleming, 'Functions of Several Variables' until get to the exterior algebra chapter at which time, if care, can now get the thin

   Henri Cartan, {\it Differential Forms,\/} ISBN 0-486-45010-4, Dover, Mineola, NY, 2006.\ \

   in English.

   Between Halmos, baby Rudin, and Spivak, you will have covered Harvard's Math 55 with a colorful description at

   http://www.american.com/archive/2008/march-april-magazine-co...

   Harvard tries to cover these three for freshman, but in most math departments the material will take you through all or nearly all of a focused undergraduate pure math major.

   If somewhere take a course in abstract algebra, e.g., with a little group theory, then that might help!

   Graduate School. Might learn a little more about topology, say, from Simmons, 'Introduction to Topology and Modern Analysis'. So, get good with metric spaces and get started on duality.

   The next big step is a course in measure theory and functional analysis. The Simmons work will help. Baby Rudin will be crucial; Halmos is recommended. So, with measure theory, do calculus over again and in a very different and much more powerful way and a way just crucial, even central, for mathematical approaches to statistics. The functional analysis will concentrate on representation theorems, the Radon-Nikodym theorem, and Hilbert and Banach spaces. Long popular, from Stanford and sometimes aimed at statistics students, is Royden, 'Real Analysis'. It's gorgeous. Should also read the real half of Rudin, 'Real and Complex Analysis'; it's a few steps up in difficulty from baby Rudin. Again, hint: Get a good course from a good teacher who can get you over the material without getting stuck. Then go back and study the material 2-3 more times, apply it, do some original research in it, and finally begin to understand it.

   We're talking high, top, center crown jewels of civilization here; the stuff is of just awesome power; my view is that it is, for the rest of this century, one of the main pillars of increases in economic productivity via the exploitation of Moore's law; on 'what to program', computer science is stuck and this material is the most promising way forward; as a famous restaurant owner once said about some Morey St. Denis, "you won't find better".

   Now are ready for probability. I recommend:

   Leo Breiman, {\it Probability,\/} ISBN 0-89871-296-3, SIAM, Philadelphia.\ \

   M.\ Lo\`eve, {\it Probability Theory, I and II, 4th Edition,\/} Springer-Verlag, New York.\ \

   Kai Lai Chung, {\it A Course in Probability Theory, Second Edition,\/} ISBN 0-12-174650-X, Academic Press, New York.\ \

   Jacques Neveu, {\it Mathematical Foundations of the Calculus of Probability,\/} Holden-Day, San Francisco.\ \

   Neveu is succinct, gorgeous, but not easy. This material is NOT popular in US departments of mathematics. At Princeton, see Cinlar.

   Then should make some progress with stochastic processes: The big book is Gihman and and Skorohod, right, in three volumes, but mostly people settle for shorter treatments. Whatever, should learn about Poisson processes, Markov processes (discrete time, finite state space is enough to get started), Brownian motion, and martingales. Might also learn about second order stationary processes. A good course in stochastic processes is NOT easy to find, especially in mathematics departments.

   Now are ready to attack statistics mathematically! I don't know of a good, single 'mathematical statistics' book at this level. Instead, there are many books -- I gave some above -- and then the journals. Thankfully, the field is relatively close to applications; so can take a practical problem and concentrate on what is relevant to it. One of my papers was some new work, at this level, in mathematical statistics for a problem in practical computing and computer science. The computer science community struggled terribly with the mathematics. So, it was some progress in computer science that community will have to struggle to understand.

   One approach to work in computing is just to try things, that is, just to throw things against the wall and see if they appear to stick. Or, maybe the truth of the situation really is a simple statistical model. Likely that model will fit the data well. So, try many simple statistical models. We use these models mostly ignoring the mathematical assumptions; mostly we are proceeding 'heuristically', that is, with guessing. If any of the models fit well, then they can be considered candidates for the truth. So, are throwing things against the wall to see if they fit. This approach also called 'data mining'.

   Problems:

   (1) Will be quite limited in what statistical models can use. That is, will be drawing from a cookbook instead of being a real chef who can create good, new dishes appropriate for the available ingredients and customers!

   (2) Don't have much, e.g., have not proceeded mathematically where from the deductive logic of assumptions and proofs actually know in advance some good things about the results. Something like breaking into a pharmacy, mixing up a lot of pills, taking them, and seeing if feel better! Uh, I'll pass and let you do that without me!

   (3) May have gone through a lot of computer time in an 'exponential, combinatorial explosion' of efforts throwing against the wall.

   (4) Have ignored a LOT in statistics that can add to what know about the results.

   (5) Will be tempted to conclude have have found 'causality' but will likely not have.

   (6) Will be tempted to conclude that have a model that predicts, but that is on shaky ground and risky and needs more work.

   Applied to important problems, this approach can be dangerous.

   There are not many healthy statistics departments. Much of the career interest is in biostatistics, especially related to FDA rules.

   It appears that among the top statistics departments are Berkeley and UNC. Since Breiman and Brillinger are at Berkeley and since Stanford, long good in statistics, is not far away, if I were looking for a Ph.D. in statistics then I'd pick Berkeley.

   There is a general problem getting a 'job' in a technical field and likely also with statistics. The assumption in US business is still as in factories 150 years ago: The supervisor knows more than the subordinate; the subordinate is supposed just to add common labor to what the supervisor says. In particular job descriptions are written by the supervisors, not the subordinates!

   Well, there are nearly no supervisors in US business who have even as much as a weak little hollow hint of a tiny clue about the material described here. So, won't need that material to qualify for the job descriptions. Moreover, if actually know such material and let that fact leak out, then will likely not make it past the first level HR English major phone screen person who will tremble and conclude that you are not like the employees they have! If you do get hired and someone in your management chain discovers that you have used some mathematics they don't understand, you might be on the way out the door, especially is your work was valuable for the company!

   Of course, the solution is to find a valuable application and start your own business. While maybe biomedical venture capital can understand crucial, core technical content, in information technology venture capital, likely you will be trying to explain this stuff to, say, history majors who worked in fund raising, marketing, general management, or financial analysis or have a background in just relatively elementary parts of computing. Just will NOT find more than six people, maybe not more than zero people, in US venture capital who can work the exercises in Royden or explain the strong law of large numbers. Sorry 'bout that! So, if you explain that the value of your venture is the powerful material in your 'secret sauce', then you will be regarded as a kook, far outside the mainstream of venture funded entrepreneurs, discarded, maybe even laughed at. As it is, some of the venture people are making money now, and the rest just want to be more like the ones who are making money. Looking for anything really new, powerful, and valuable is just NOT in the picture.

   So, once you have some results in users, customers, revenue, etc., then maybe you can get some venture funding; just why at that point, owning 100% of the business, you would take venture funding, a Board that can fire you, etc. is less clear! Or venture funding is not for everyone! Or venture firms prefer to give money to people who don't need it!

   For the real power of the 'secret sauce', you have just to keep that a secret!

   Once mathematicians have yachts, at the venture firms math will be to info tech like biochem is to biotech. In the meanwhile note that a valuable application of statistics can put you on the Forbes 400 where there are not many people! Generally if you are making a valuable application of advanced or new statistics, then you will not know many people who understand what you are doing. Or, if lots of people understood it, then it wouldn't be valuable!

   Show morethread