Name: Measure theory
ISBN: 9780387900889

What Hacker News says

graycatOct 2015

No, he's just being sloppy.

That can be a real pain for students trying to learn. But, be warned: A lot of people work with probability, but only a tiny fraction ever had a course in graduate probability. So, eventually have to learn to put up with, and sometimes rewrite, some of what is written that is not very precise.

Here are the accepted definitions:

Take a non-empty set, usually denoted by Omega, and call it the sample space or the set of trials.

Take a collection of subsets of Omega, usually denoted by script F, and call it the set of events. Have at least enough subsets of Omega in the set of events script F so that script F will be a sigma algebra. So, script F has to have as an element the empty set, be closed under relative complements, and be closed under countable unions.

We want all that for probability theory. We want countably infinite so that we can discuss, say, the event that a coin never or always comes up heads. We don't want uncountably infinite because it would create a big mess in the theory.

Then the ordered pair (Omega, script F) is called a measurable space -- it doesn't have a measure yet but soon will.

On that measurable space, define a positive measure P so that P( Omega ) = 1. Then P is a probability measure on the measurable space.

A probability space is the triple ( Omega, script F, P ).

Intuitively, a measure assigns to each event A in script F a non-negative real number P(A). A signed measure permits negative values. And of course at times want a measure that yields complex numbers.

Measure theory is in, say, P. Halmos, Measure Theory. The back of Halmos has a really nice introduction to probability theory, the Kolmogorov three series theorem in stochastic processes, etc. For more in stochastic processes there is, of course, the classic J. Doob, Stochastic Processes. Halmos was a Doob student and then served as an assistant to von Neumann at the Institute for Advanced Study. From that he wrote Halmos, Finite Dimensional Vector Spaces, a finite dimensional introduction to Hilbert space. Later at University of Chicago Halmos worked in mathematical statistics and made the fundamental contribution to sufficient statistics. IIRC, it was at Chicago that Halmos wrote Measure Theory.

Great details on measure theory and much more are in, say, H. Royden, Real Analysis and W. Rudin, Real and Complex Analysis.

To narrow measure theory to probability theory, use the famous texts by the authors I listed, Loeve, .... Royden and Rudin are terrific prerequisites to Loeve, etc.

Commonly the sigma algebra of events script F is the smallest sigma algebra that contains as a subset a topology, that is, a collection of open sets, on the sample space Omega.

That's all standard, beginning stuff in graduate probability -- ah, the only kind should bother with anyway!

thread

graycatJul 2015

Halmos Measure Theory is a good prerequisite to Neveu. Rudin, Principles is a bit too little. Instead, the first half, the real half of Rudin's Real and Complex Analysis is a good prerequisite. So, is Royden's Real Analysis.

Neveu is elegant beyond belief, but Breiman, Probability, the SIAM book, available in paperback, is darned good, usually easier than Neveu, less elegant, closer to applications, and without some of the special Tulcea material in the back of Neveu. K. L. Chung also has a good, comparable book. Even if want Neveu to be your main probability book, which is fine, likely you should have alternative treatments.

Of course, there is Loeve, Probability -- written in English but somehow sounding like French. It has a lot, a little too much, but I liked the topics I studied in it. It turns out, Neveu and Breiman were both Loeve students.

Halmos, Measure Theory, is darned fun to read: It has the three series theorem and a famous exercise on regular conditional probabilities.

I learned the stuff from a course by A. Karr, a star student of E. Cinlar. Karr's course was the best course of any kind I ever took in school. Powerful material, beautifully presented, each day it was a shame to erase the board.

The exercises in Neveu are usually harder than the ones in Halmos, Breiman, and Chung.

Neveu makes probability a crown jewel of civilization.

The summer after Karr's course, I sat in the library for six weeks and walked out with a 50 page manuscript that was all the research and the first draft of my dissertation. Net, probability at the level of Neveu is darned powerful stuff, makes a lot in research, and research for applications, really easy -- that is, you really know just what the heck you are doing and can knock off new results having fun sitting in bed next to your wife while she watches TV (warning -- not gender neutral!).

What I've outlined is sometimes just called graduate probability. The biggest difference is that the whole subject makes daily use of measure theory.

I don't know how much you need in probability before starting on graduate probability. In my case, graduate probability was my first serious study of probability, and I never felt that I was not prepared.

But in my career I'd done a lot of practical work in both probability and statistics -- e.g., multivariate statistics, hypothesis testing, stochastic processes, digital filtering, the fast Fourier transform, beam forming (a case of antenna theory), power spectral estimation (US Navy sonar type stuff), how to get the central limit theorem out of digital filtering, and more, random number generation, etc. That work was plenty of intuitive background for graduate probability.

But in much of that work I was struggling due to what, really, at that level, is commonly weak basic knowledge of probability. So, after those struggles, seeing graduate probability be all clean and powerful was great.

I can't advise on just how much elementary probability you might need to have enough intuition to be comfortable with graduate probability. I will say, you do need both the intuitive experience and also the solid math.

I feel sorry for people who work in prob/stat without a background in grad prob: The elementary stuff is too often just confused from poor understanding from a poor background.

The sources I mentioned above were really the first sources from which I did any real study. Net, the elementary material of prob/stat is really too simple to be taken very seriously. So, for your first serious effort, just go for graduate probability from the sources above.

The Neveu, etc., material is much of the foundation for the secret sauce of my startup.

thread

caesiumOct 2010

Rosenlicht. Linear Algbera done Right. Then Finite Dimensional Vector Spaces. Halmos' Measure Theory. Awesome.

thread

graycatJul 2018

For ergodic theory, there were a few really good lectures, and I have good notes, in a course I took from a star student of E. Cinlar.

For Breiman Probability:

(1) I liked his start, nice, simple, intuitive, on how the law of large numbers works. Of course the super nice proof is from the martingale convergence theorem, but Breiman's start is nice.

(2) His book has the nicest start on diffusions I have seen.

(3) Once in some of my work I ran into a martingale. So, then, when I wanted to review martingale theory, I went to Breiman. Nice.

(4) Since I was interested in stochastic optimal control, I was interested in measurable selection and regular conditional distributions, and I got those from Breiman -- thanks Leo!!!! Yes, IIRC, there is the relevant Sierpinski exercise in Halmos, Measure Theory and also in Loeve, Probability, but it was Breiman who gave me what I really needed.

(5) Generally Breiman is easy to read. He writes like he is trying to help the reader learn.

thread

graycatJun 2017

Part II

With good stuff in ML you are describing a world I've seen no solid evidence of first hand.

Maybe that's to be expected: Long I did well learning math in part because I picked some really good books; due to some lucky parts of my background, that was easy to do. Then I tried this in statistics and picked a junk book -- eventually figured that out.

I didn't have much respect for or progress in statistics before I had a high end background in probability from a star student of E. Cinlar at Princeton from Neveu's book, Neveu a star student of Loeve at Berkeley. Then in stochastic processes I got some more respect from work of J. Doob (prof of Halmos who then became an assistant to von Neumann, etc.) and the material on probability in the back of the Halmos, Measure Theory. The Halmos work on sufficient statistics and unbiased estimation also deserved respect.

I've seen no such quality from ML. You suggest that there is such quality out there, but what I saw, e.g., Ng's on-line lectures on ML, the idea of convolutional neural networks for image recognition, doesn't look nearly as solid as what I did in probability or nearly as interesting as some of what you point to.

E.g., just yesterday I went to

https://simplystatistics.org/

a Web site of three profs in bio-stat, and read some of the material of one of them, Jeff Leek, looked at the TOC and some of the lectures in his Coursera on-line course, some of the research problems he's working on, some of what he said about ML, what he said about cross-validation and bootsrapping in model building, etc. and concluded that with ML still I was not missing much.

I did look at

Trevor Hastie, Robert Tibshirani, and Jerome Friedman, {\it The Elements of Statistical Learning Data:\ \ Mining, Inference, and Prediction, Second {\it Edition,\/} Springer, 2008.\ \

found the interesting looking work on over fitting and otherwise was not impressed at all.

Yes, you have more references I have yet to pursue, but what I did yesterday and other times was fast and easy and seemed to touch on some of the most respected work. But, yes, it took me some real digging and a lot of luck to find the really good stuff in probability, stochastic processes, optimization, etc., so maybe I'd need a lot of digging to find the good stuff in ML. Since ML is a more recent field, likely it has yet to receive the polish of Tukey, Halmos, Loeve, Neveu, Doob, Dynkin, Shreve, even back to Kolmogorov, etc.

I do wonder, then, what the heck the ML people are driving at? Really it looks like they are growing around the edges of old regression analysis model building, trying to fit to data (training data) and then use the fit to make predictions. Okay. But I'm not going to take 5 TB of pictures and try to recognize cats, dogs, men, women, etc. with some version of curve fitting with many thousands of variables -- that's just a long way from anything I'm doing, and more generally I'm reluctant to believe that any technique that needs that much data will be more than a niche. E.g., for intelligence, kittens, puppies, and toddlers do really well with much less data. E.g., a kitten who jumps up on the kitchen counter, walks over to the stove, and gets a hot foot learns well from just that one data point. Moreover he starts to find causality and generalize.

Really, my view is that for more in getting intelligence out of data, we should concentrate on cases where there likely is some version of causality and have our applied math look for and then use that. I didn't see ML doing that. My brother and my wife got their Ph.D. degrees in the social sciences, and there they were quite careful about causality.

For more parameters than data points, okay: Sure, in some non-linear cases, that might happen. And even in linear cases, if just open eyes a little, can see that and, thus, get around over fitting in some cases. E.g., regression is really a perpendicular projection onto a linear subspace spanned by the vectors of the input data. Okay. Now just enter the input data twice. So, then the dats still spans the same linear subspace and we get the same perpendicular projections. If write out the regression normal equations, then the variance-covariance matrix get won't be non-singular and won't have an inverse. So, big heart burn. But heart burn is premature: The normal equations still have a solution, just infinitely many. So, the regression coefficients aren't unique. BUT, and apparently not many people have noticed this, all the predicted values from the given data still are unique, that is, are just the same projection. That is, take any of the infinitely many solutions you want from the normal equations and get the same predicted values from the given data (maybe not new data). That observation solves some of over fitting but not all. But for your remark, that IS a case of working with too many parameters.

For more, looking at Jeff Leek on regression model building, he quickly goes into cross validation and bootstrap techniques. Okay, cross validation looks like intuitive, heuristic, Medieval stirring pots of rat tails and frog eyes to me, and boostrap needs some theorems (I believe I have derived some -- I know, look at Bradley Efron's work which I've done a little but so far I prefer my derivations) to justify and explain what is going on, but roughly Leek is correct, and I didn't see even that much care and concern in the ML materials.

If ML is still trying to milk some more utility out of the edges of simple, old regression, that is, fitting to data and using the fits to predict, then I'm not much interested: Often regression works really well; where it doesn't, and IMHO over half the time it doesn't, then I'm not going to strain to squeeze a little more out of that curve fitting paradigm, at least not now.

To me, curve fitting to build predictive models is nothing like all that can be done with statistics and only a tiny drop of what can be done with applied math.

E.g., where is the good science from simple, linear equations? From Newton, Maxwell, Einstein? Nope, none of those. Where did they make progress with fitting linear equations? Gee, Kepler used ellipses and Newton drew from those.

For fitting, the big success was Ptolemy's epi-cycles, and that still was just fitting and not causality or real science or physics.

Moreover, now I'm concentrating on my startup -- I've long since derived the math, and I see nothing in ML to even hint that their work could improve on mine.

Sure, when my startup is done, I'll take on new interests. But curve fitting? I doubt it! Real AI? Maybe. More likely physics!

Sure, maybe some day there will be a Kolmogorov, Doob, etc. to do terrific work in ML or new frontiers of statistics and/or applied math and a Loeve/Neveu to do a polished job writing up the work. Maybe.

Then there's a credibility problem: From what I've seen, nearly none of the computer science profs working in AL/ML have what Jeff Leek called "math chops" that promise they can do really good research to move statistics ahead.

I'd have a sign over the door reading "None enter without a proof that there are no countably infinite sigma algebras."!

That will select the people with good "math chops".

From your claims, maybe there actually is some good work in ML but hard to find; as I started, hard to find would not be too surprising since early in my career finding the good stuff in probability and optimization was really hard. Even in the statistics community, the good people are hard to find. I can respect E. Dynkin, E. Cinlar, J. Tukey, L. Breiman, D. Brillinger, and a few more, but for a beginner in statistics those people will be difficult to find, understand, appreciate, respect, etc. Maybe something similar is the case in ML now.

In statistics taken broadly, there is a guy from France I know and respect; he was a student of Choquet, right, a member of Bourbaki!

But, again, for ML to have my respect, tough if they are just still trying to do curve fitting.

thread

graycatMar 2014

> Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed.

No. I defined sigma, that is, standard deviation, fully precisely and correctly. The normal distribution has nothing to do with that definition. And the standard deviation is a number, just a number, just as I defined it as

σ = E[(X - E[X])^2]^(1/2)

which clearly is just a number and not an area.

Or, for random variable X with cumulative distribution F_X, that is, for real number x,

P(X <= x) = F_X(x)

we have, with notation from D. Knuth's TeX, that

σ^2 = \int (x - E[X])^2 dF_X(x)

This integral need not be in the sense of Riemann (i.e., freshman calculus) because dF_X is a measure on the real line; so, the integral is in the sense of measure theory (see any of Rudin, Real and Complex Analysis; Royden, Real Analysis; Halmos, Measure Theory; Loève, Probability Theory).

Sigma is defined for any random variable X or its distribution provided that E[X] exists and is finite. Again, a "normal" or Gaussian assumption is not necessary. So, sigma is defined for discrete distributions, the uniform distribution, the Poisson distribution, the exponential distribution, etc.

For a random variable X, if don't know its distribution, then can't say what the numerical value of its standard distribution is.

Moreover if for some random variable X that has a standard deviation want to know, say, the probability

P( -5σ <= X <= 5σ)

then that is an area and need the distribution of X to find the numerical value.

All that is 100%, completely, totally, absolutely true. That's what σ or standard deviation is.

In particular, the statement

"In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)."

is flatly false. The field of statistics has no such statement or convention.

Yes, if want to be really sloppy, make some assumptions not clearly stated, and have some conventions for identifying some things in special ways, e.g., that sigma is an area, then can do so. Maybe some parts of physics do this. I do remember when I was studying physics the prof handed out a little book on how errors were handled in physics. The book was a sloppy mess and one of the reasons I lost respect for accuracy and precision in physics and majored in math instead.

It is true that about 100 years ago some fields of study, especially parts of psychology and much of education, concluded that the Gaussian distribution was some universal law of data handed down by God. Well, God did no such thing. Still, some people in educational statistics believe that student test scores should have a Gaussian distribution and, if the scores do not have such a distribution, will, from many such scores, find the empirical distribution and, then, transform the scores so that the distribution is closely Gaussian.

Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get

"68% of the data is within one standard deviation of the mean, 95% is within two, and so on.",

and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking and not mathematics, probability, statistics, or anything from God.

Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value. Instead, for Gaussian random variable X with E[X} = 0 each of

P( -σ <= X <= σ)

P( -2σ <= X <= 2σ)

P( -5σ <= X <= 5σ)

has a particular numerical value.

Sorry 'bout that.

thread