Probability

What Hacker News says

1. graycatJul 2018
   To make the study easier and better rounded, just add some material.

   Rudin's Principles is really nice, especially in retrospect once understand it, but going in as a student it can seem quite severe. It's precise but not too severe -- he just makes you go a chapter or two before it becomes clear why he is doing what he is doing.

   To help, my nutshell view is that mostly he is just trying to develop the Riemann (Stieltjes) integral. His main result is, the Riemann integral exists for continuous functions on compact sets. So, then, he needs to say what a compact set is. Well, the most relevant example is just a closed interval of real numbers such as [a,b]. So, why compact? Because every continuous function on a compact set is uniformly continuous, and that lets us know that the Riemann sums converge. What is compact? Every open cover has a finite subcover, and that lets us get uniform continuity. And, in R^n, a set is compact if and only if it is closed and bounded. So, Rudin needs to talk about closed versus open -- he does that on metric spaces although really he needs it only on R^n.

   So, net, he starts with metric spaces and discusses open, closed, and compact. Then he shows that in R^n, compact is the same as closed and bounded. He shows that a continuous function on a compact set is uniformly continuous. Then, presto, he shows that the Riemann (or Riemann-Stieltjes if you wish) sums converge and the Riemann integral exists.

   He does some nice work on infinite sequences and series, and the main reason is that he uses those tools to show lots of limits exist, e.g., for sines, cosines, and Fourier series.

   There's more of value in Principles, but IMHO I gave you a good start to make the book easier. I wish I'd had been given that outline when I was working through Principles at 1+ hour a page.

   But the Lebesgue integral in Royden is the one to take fully seriously.

   Make Halmos the second or third text on linear algebra. And then look at some quantum mechanics where they discuss eigenvalues, eigenvectors, Hermitian, unitary, and the spectral decomposition! Right, the Halmos book is baby Hilbert space. Then look at some applied connections, e.g.,

   George E.\ Forsythe and Cleve B.\ Moler, {\it Computer Solution of Linear Algebraic Systems,\/}

   Maybe spend an evening on the documentation of LINPACK.

   Some weekend of great fun, take a fast pass through the Gauss, ..., Stokes theorem parts of

   Tom M.\ Apostol, {\it Mathematical Analysis: A Modern Approach to Advanced Calculus,\/}

   where don't take the proofs very seriously but to see how physical science and engineering look at calculus of several variables.

   Neveu is a great last probability text but not a good first text. So, before Neveu, look quickly, not very seriously, at whatever, including in some introductory statistics texts.

   Also, Breiman's Probability is easier to read than Neveu. So, is K. L. Chung's competitive book. And there are others.

   There is some more advanced material, e.g.,

   Ioannis Karatzas and Steven E.\ Shreve, {\it Brownian Motion and Stochastic Calculus, Second Edition,\/}

   Good luck!

   Show morethread
2. davidsrosenbergJul 2018
   Hehe ok —- I also love Breiman’s Probability book. It’s really a standout on Ergodic theory. And Breiman et al.’s book on Trees is surprisingly rich, talking about all sorts of stuff besides trees.
   thread
3. graycatJul 2018
   For ergodic theory, there were a few really good lectures, and I have good notes, in a course I took from a star student of E. Cinlar.

   For Breiman Probability:

   (1) I liked his start, nice, simple, intuitive, on how the law of large numbers works. Of course the super nice proof is from the martingale convergence theorem, but Breiman's start is nice.

   (2) His book has the nicest start on diffusions I have seen.

   (3) Once in some of my work I ran into a martingale. So, then, when I wanted to review martingale theory, I went to Breiman. Nice.

   (4) Since I was interested in stochastic optimal control, I was interested in measurable selection and regular conditional distributions, and I got those from Breiman -- thanks Leo!!!! Yes, IIRC, there is the relevant Sierpinski exercise in Halmos, Measure Theory and also in Loeve, Probability, but it was Breiman who gave me what I really needed.

   (5) Generally Breiman is easy to read. He writes like he is trying to help the reader learn.

   Show morethread
4. graycatMay 2018
   I've been a STEM field student and prof, and I've published some quite pure math research and also some AI research. My Ph.D. dissertation had its motivation from practice, e.g., from when I was Director of Operations Research at FedEx, and was an early case of what is now a major theme of the Department of Operations Research and Financial Engineering (ORFE) at Princeton. And since my Ph.D., I've made practical applications of math where the key was some relatively pure math research. Moreover, the crucial core of my intended to be fully practical startup is some applied math with some advanced pure math prerequisites.

   So, my experience is that what I suggested is not a danger but motivation and stimulation of a lot in pure research.

   Having some students do some work with people from outside academics is crucial for their professional development. Of course the work will have graduate student and faculty supervision and high quality. The students should welcome the business world contacts.

   There is a lot of the very best pure medical research in labs not far from the wards with dying patients. In important senses, that there are real patients there with their lives literally depending on the results of the research helps both the patients and the research.

   It is very much a fact of life in life, applications, and also pure research that a lot of good motivation is from good to excellent. In particular in research, for the younger researchers, finding good problems to work on is one of their most severe struggles. Well, in medicine, a young physician in a research-teaching hospital everyday can see, does see, patients dying in the wards, and that work can help the researcher find good problems to work on.

   For your

   > Whenever it makes sense to ground research in practice, CMU professors generally do so by working with industry and govt collaborators.

   "Whenever", quite commonly and generally and no exceptions? Amazing. I'm thrilled. Good for CMU. Since one of my Ph.D. dissertation advisors was long President at CMU, maybe he was in part responsible for this amazing, thrilling situation?

   Color me skeptical: My long experience in and around research tells me that pure research needs much more contact with and stimulation and motivation from practice. Yes, some pure researchers have found really good pure research problems and directions, and, then, they should continue on, but much more common is what I explained, using practical problems as motivation and stimulation, and justification, for research that might be, in my experience often can become, nicely general and pure. Or, if the research is all just routine, then pass the problem off to a ugrad for a class exercise, term paper, or senior honors paper. Else push forward for better results and encounter some real research problems.

   Here is a big example: During WWII, G. Dantzig was working on military logistics, e.g., what to ship where, how, when to aid the war effort. After the war, at RAND for, IIRC, the USAF, he continued and as a first cut invented linear programming. About then a special case of that, for the transportation, problem, the "translocation of masses" resulted in a Nobel prize in economics for L. Kantorovich. So, linear programming was already making progress enough in pure research to yield a Nobel prize in economics. And there were some more Nobel prizes from linear programming and optimization.

   To solve linear programming problems, Dantzig invented his simplex algorithm, basically a nice tweak on Gauss elimination for systems of linear equations. Cute. Not very pure but at one time rated as one of the most important pieces of work in engineering of the 20th century. In practice, nearly always it is shockingly fast, and it took some nice work decades later in some relatively pure math of computational geometry (K. Borgwardt) to show why it was so fast.

   Continuing on, soon enough it was observed that, often in practice, it was required that the variables be restricted to whole number values. That is, the real, practical problems were often integer linear programming (ILP). IIRC, first cut, Dantzig expected that a tweak of his simplex algorithm would be able to handle that.

   Work on ILP continued, for decades. There were lots of important practical problems for motivation, e.g., network design at Bell Labs. That problem has remained important, e.g., was the subject of an A. Goldman lecture at Johns Hopkins by MIT Dean of Science T. Magnanti. And there was progress on solutions.

   ILP was taken seriously by, okay, Princeton grad R. Gomory.

   By then computer science had discovered the problem of sorting, saw that simple bubble sort ran in O(n^2) but that heap sort ran in worst case and average case O(n ln(n)) and met the Gleason bound and, thus, was in the sense of big-O the fastest possible sorting algorithm for sorting just by comparing pairs of keys. So, this was progress in computational time complexity. Since heap sort is also in-place, it was also progress in computational space complexity.

   So, with both the practical successes and the struggles of ILP and the practical success of the simplex algorithm and the specter of O( e^n ) for ILP, there was the serious research question of what would be the fastest algorithm in worst case for ILP. This question was asked and explored at Bell Labs and resulted in the now famous

   Michael R. Garey and David S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, ISBN 0-7167-1045-5, W. H. Freeman, San Francisco, 1979.

   So, from there ILP is, yes, in NP-complete.

   So, now we have at Clay Math in Boston a prize of $1 million for the first solution of the problem in computational time complexity of P versus NP, generally considered one of the most important problems in both pure and applied math and computer science.

   Lesson: Practical problems, taken seriously, can result in some of the most important problems in pure research, and some of the progress in pure research can help get solutions to some practical problems. The motivation from pressing practical problems can help drive the research in both pure and applied research.

   In particular, the OP was about CMU, CS, and AI. From what I've seen and heard about AI, a lot of what is of interest now, and likely a big part of the CMU AI ugrad program, is "modern regression analysis". Maybe CS and AI need modern here because otherwise they are open to accusations of reinventing and pushing out a lot of hype about some multivariate statistics quite mature as math 50+ years ago.

   If CMU CS and AI are willing to take regression so seriously, also going for some of what I mentioned, e.g., convex programming, stochastic optimal control, should be regarded as much more worthy. Making stochastic optimal control more practical is one heck of a challenge but with some progress possible, e.g., as now at the ORFE Department at Princeton.

   And we should note that much of the AI interest in regression is based on the work of L. Breiman in Classification and Regression Trees (CART). Breiman was, IIRC "an academic probabilist"; his text Probability (one of my favorites, e.g., for measurable selection) was all based heavily on measure theory; and his work on CART started by trying to get fits and predictive models starting with complicated data from clinical medicine. So, here again, some pressing practical problems in practical medicine led Breiman to CART which is now one of the main pillars of AI. Given that background, the CMU CS AI program should welcome the level of contact with real problems I described without your concern about the death of pure research.

   Show morethread
5. graycatApr 2018
   In most academic fields, work that mathematizes the field is regarded as the best.

   Neveu is elegant beyond belief. I keep my copy close. I was aimed at Neveu by a star student of E. Cinlar, long at Princeton and before that at Northwestern -- long editor in chief of Mathematics of Operations Research. Neveu was a student of M. Loeve at Berkeley. So was the current darling of machine learning, L. Breiman, because of his Classification and Regression Trees (CART). Breiman's Probability as published by SIAM is generally easier reading than Neveu.

   For stochastic processes, there are several relatively different directions to go.

   Martingale theory is gorgeous, astounding, amazing, with one of the most powerful inequalities in math, the astounding, tough to believe, martingale convergence theorem, and likely the shortest proof of the strong law of large numbers.

   Then can do Markov processes more generally. The discrete state space version is important and not too difficult -- Cinlar has a nice introductory text.

   A high end direction for Markov processes is potential theory. There are claims that that is the math for exotic options on Wall Street, but I doubt that there have ever been any applications.

   There is a big role for second order stationary stochastic processes in electronic engineering. I ran into that for processing ocean wave data for the US Navy. Here the fast Fourier transform added a lot of interest.

   And there's more.

   Generally long Russia, France, and Japan seemed to have emphasized stochastic processes more than the US. But by now I suspect that the US is well caught up.

   I'd have a tough time believing that very many people with money to hire know enough about high end stochastic processes, or even just Neveu, to hire in those fields. US national security may be about the only hope, that is, outside of academics.

   Yes it appears that some of the quantum field theorists in physics are interested in path integrals.

   Uh, I'm disorganized here: There is the field of stochastic optimal control!

   As usual for advanced applied math, my suggestion is, outside of academics or US national security, find a valuable application and start a business to make money. That is, don't expect to be hired.

   Show morethread
6. graycatAug 2014
   > People have been building on top of the axiomatic foundation that you outlined for decades. This is old news.

   Yup, essentially since Kolmogorov's 1933 paper.

   The axiomatic approach I gave is just what is used in Breiman's Probability, as published by SIAM. It's a super nicely written book. I have never seen his book on statistics.

   thread