Stats for nerds
3recommend
30mentions
6people
2009–2025years
63% recommend37% neutral
Mentions per year
What Hacker News says
- "Functions Are Vectors"?
Let's see: Let A be a non-empty set, N the set of positive whole numbers, and X the set of all functions f
f: A --> N
with usual notation.
Assume as is common, the scalars are the set of real numbers, but the set of complex numbers will also do.
So, is X a vector space and, thus, each f in X a vector?
No, since -f is not in X. Neither is (1/2)f.
Some references (with TeX markup):
Paul R.\ Halmos, {\it Finite-Dimensional Vector Spaces, Second Edition\/}
linear algebra treated as functional analysis.
Walter Rudin, {\it Real and Complex Analysis\/}
with Lebesgue integration and, then, Banach and Hilbert vector spaces.
Walter Rudin, {\it Functional Analysis\/}
with Fourier theory.
Jacques Neveu, {\it Mathematical Foundations of the Calculus of Probability\/}
with random variables, that is, functions from a probability space to, usually, the set of real numbers with convergence results, building on the work A. Kolmogorov building on the work of H. Lebesgue.
- > The unreasonable effectiveness of conditional probabilities
Yup.
There is a more general result that is more amazing with still more "unreasonable effectiveness":
For events A and B, we can consider the conditional probability of A given B:
P(A|B) = P(A and B)/P(B)
Then we can generalize P(A|B):
Suppose we do an experiment and make two observations, X and Y. We regard each of X and Y as a random variable that takes on real number values.
We assume that X and Y have averages, that is, expectations, and write those as E[X] for the expectation of X and E[Y] for the expectation of Y.
We want to use X to approximate Y.
So, we want a real valued function f of one real variable x and use f(X) to approximate Y.
For the most accurate approximation, we want to minimize the expectation
E[(Y - f(X))^2]
Claim: For f(X) we want
f(X) = E[Y|X]
where E[Y}X} is the conditional expectation of Y given X and a generalization of P(A|B).
Proof:
We start by using one of the properties of conditional expectation and then continue with just simple algebra:
E[(Y - f(X))^2]
= E[ E[(Y - f(X))^2|X] ]
= E[ E[Y^2 - 2Yf(X) + f(X)^2|X] ]
= E[ E[Y^2|X] - 2f(X)E[Y|X] + f(X)^2 ]
= E[ E[Y^2|X] + E[Y|X]^2 - 2f(X)E[Y|X]
+ f(X)^2 - E[Y|X]^2 ]
= E[ E[Y^2|X] + ( E[Y|X] - f(X) )^2 - E[Y|X]^2 ]
which is minimized with
f(X) = E[Y|X]
Done.
So, the "unreasonable effectiveness" of conditional expectation that it provides the best non-linear least squares approximation.
In terms of statistics E[Y|X] is the minimum variance estimator of Y and is also unbiased.
Yes, P(A|B), conditional probability, can be seen as a special case of E[Y|X], conditional expectation.
The detailed definition of E[Y|X] is based on the Radon-Nikodym theorem. W. Rudin's Real and Complex Analysis gives the novel von Neumann proof. Then in Neveu, Mathematical Foundations of the Calculus of Probability is elegant coverage of E[Y|X].
- Here we see another problem, the use of random. At the level of Kolmogorov, Loeve, Breiman, Neveu, Cinlar, Karr, etc., we use random only in random variable. We define random variable carefully but never say what random by itself means.
The MIT quantum mechanics lecture I mentioned keeps using random -- struggles with it, says it over and over as if that would make it more clear. In that lecture, the meaning of random is not clear.
What that prof means is likely
"the values of some random variables that are independent and identically distributed (i.i.d.)."
Then we can have some random variables that take values only on the positive, even integers. But, sure, they can't be uniformly distributed. So, with the values of such random variables, we can get some even integers selected independently.
In good treatments of probability, e.g., (from my bibliography file, with TeX markup)
Jacques Neveu, {\it Mathematical Foundations of the Calculus of Probability,\/} Holden-Day, San Francisco.
independence is very nicely developed, e.g., to handle even a set of uncountably infinitely many independent random variables.
As I recall, Neveu was a Loeve student.
That book my Neveu is my candidate for the most elegantly written math book of all time.
But the development of independence is not trivial.
I would expect that independence is needed at times in physics, especially in quantum mechanics, but that only a small fraction of physics profs have studied independence much like what is in Neveu. One result is some confusion between independent and uncorrelated. When I'm watching a physics lecture and see such confusion, I stop taking the physics prof seriously and turn to something more serious, say, an old Marilyn Monroe movie!
- Uh, the OP frequently mentions "statistical independence", and I am uncomfortable with that: Independence is a property of a set of more than one random variable and is defined in probability, often used in statistics, but not defined in statistics. There is a polished, elegant, and thorough treatment of such independence, including of uncountably infinite sets of random variables, in J. Neveu, Mathematical Foundations of the Calculus of Probability.
- My view, from math, computing, and computer science is (1) that the author is being too critical and often throwing out a small but good baby with some not very dirty bathwater and (2) that he is not mentioning some larger problems, worse "sins".
Here would be my list of the worst sins or how to avoid them:
(1) A term is a word used with other than a standard dictionary definition. Technical fields are awash in terms. Make sure to try hard to define all the terms you use; with some judgment applied, can omit definitions of some terms certainly well understood by any reader with enough background to get anything from the paper. Poorly or undefined terms can be one of the best ways to lose readers. Here, hoping to get more readers, maybe inviting them into the field, bend over backwards in defining terms, that is, maybe give some term definitions for readers who, really, have little hope of getting much from the paper -- at least, when such a reader gives up they won't blame the author!
In addition, for any term new or relatively new to the field, with some judgment, might also include for the term motivation of its importance and examples of its usage -- i.e., make clear the importance, relevance, value, usage, etc. of the term.
(2) For nearly all acronyms, standard in the field or not, for the first use of the acronym include what it abbreviates. Here to be sure are doing well on this issue, strain and bend over backwards and include, say, even TCP (transmission control protocol), SMTP (simple mail transfer protocol), CPU (central processing unit) -- sure, I'm suggesting bending over backwards. Several slaps on the wrist, an hour in the corner with wearing a dunce cap, and a coating of tar and feathers for each acronym used but not defined with what it abbreviates.
(3) For how to write math, in two parts, (A) and (B):
(A) Take some theorem proving courses from some of the most precisely written texts, say, P. Halmos, Finite Dimensional Vector Spaces, W. Rudin, Principles of Mathematical Analysis, Neveu, Mathematical Foundations of the Calculus of Probability, E. Nering, Linear Algebra and Matrix Theory, Royden, Real Analysis where do a lot of the exercises as homework and where the professor DOES read and remark on your writing. Neveu was a Loeve student at Berkeley. Nering was an Artin student at Princeton. As I recall, Rudin's background was in Austria, although I don't know who his professors were. Of course, Royden was long at Stanford. For more, if have time and insist on some really good examples, read some of Bourbaki.
By the way, on use of we, that is standard. So we might have,
"Given topological spaces X and Y and a function f: X --> Y, we say that function f is continuous provided for each set B open in Y f^(-1)(B) is open in X.
So, that's a vanilla example of using we in mathematical writing.
(B) Do some things none of those texts do: Include some intuitive views, some helpful pictures, motivation via applications in math and also outside math, and outlines of research directions.
There is a lot of question about how appropriate is suggestion (B); in math my guess is that there is no question about the relevance, wisdom, importance, value, etc. of suggestion (A).
For the goal of writing math, suggestion (A) is important: Tough to expect good success without the texts I listed or other texts written with similar care. That level of care is extreme, tough to find and learn, and much tougher to do. Writing math with the care of those texts seems to have been understood and practiced significantly often only after about 1950 or 1940. And for at least one course from at least one such text, DO have the good homework grading of a good math professor.
Personal experience and lesson: While I'm not much interested in being a professor, I have published some papers, and from getting those papers reviewed my guess would be that about half of the reason the papers passed review, and they always did with no significant revision or difficulty, is that I wrote the math with nearly the care and precision of, say, W. Rudin. Point: To critical readers, any lapse in that level of care and precision can be like a worm in a baked apple, perhaps otherwise terrific from brown sugar, butter, etc. Or, might guess that, in nearly any field, the good work is less than 10% of the total, maybe less than 1%, with the rest flawed, maybe as bad as that apple with a worm. In writing math, it is way too easy to be in the bottom 99% or 90% just from the care and precision of the writing, and suggestion (A) is IMHO (in my humble opinion) a good way to have at least the writing quality keep you in the top 90+%.
For math used in computer science, for how to write that math carefully, there are of course examples from D. Knuth.
- Part II
(2) Linear Algebra
(2.1) Linear Equations
The start of linear algebra was seen in high school algebra, solving systems of linear equations.
E.g., we seek numerical values of x and y so that
So, that is two equations in the two unknowns x and y.3 x - 2 y = 7 -x + 2 y = 8Well, for positive integers m and n, we can have m linear (linear is in the above example but omitting here a careful definition) equations in n unknowns.
Then depending on the constants, there will be none, one, or infinitely many solutions.
E.g., likely the central technique of ML and data science is fitting a linear equation to data. There the central idea is the set of normal equations which are linear (and, crucially, symmetric and non-negative semi-definite as covered carefully in linear algebra).
(2.2) Gauss Elimination
The first technique for attacking linear equations is Gauss elimination. There can determine if there are none, one, or infinitely many solutions. For one solution, can find it. For infinitely many solutions can find one solution and for the rest characterize them as from arbitrary values of several of the variables.
(2.3) Vectors and Matrices
A nice step forward in working with systems of linear equations is the subject of vectors and matrices.
A good start is just
we saw above. What we do is just rip out the x and y, call that pair a vector, leave the constants on the left as a matrix, and regard the constants on the right side as another vector. Then the left side becomes the matrix theory product of the matrix of the constants and the vector of the unknowns x and y.3 x - 2 y = 7 -x + 2 y = 8The matrix will have two rows and two columns written roughly as in
So, this matrix is said to be 2 x 2 (2 by 2)./ \ | 3 - 2 | | | | -1 2 | \ /Sure, for positive integers m and n, we can have a matrix that is m x n (m by n) which means m rows and n columns.
The vector of the unknowns x and y is 2 x 1 and is written
So, we can say that the matrix is A; the unknowns are the components of vector v; the right side is vector b; and that the system of equations is/ \ | x | | | | y | \ /
where the Av is the matrix product of A and v. How is this product defined? It is defined to give us just what we had with the equations we started with -- here omitting a careful definition.Av = bSo, we use a matrix and two vectors as new notation to write our system of linear equations. That's the start of matrix theory.
It turns out that our new notation is another pillar of civilization.
Given a m x n matrix A and an n x p matrix B, we can form the m x p matrix product AB. Amazingly, this product is associative. That is, if we have p x q matrix C then we can form m x q product
ABC = (AB)C = A(BC)
It turns out this fact is profound and powerful.
The proof is based on interchanging the order two summation signs, and that fact generalizes.
Matrix product is the first good example of a linear operator in a linear system. The world is awash in linear systems. There is a lot on linear operators, e.g., Dunford and Schwartz, Linear Operators. Electronic engineering, acoustics, and quantum mechanics are awash in linear operators.
To build a model of the real world, for ML, AL, data science, ..., etc., the obvious first cut is to build a linear system.
And if one linear system does not fit very well, then we can use several in patches of some kind.
(2.4) Vector Spaces
For the set of real numbers R and a positive integer n, consider the set V of all n x 1 vectors of real numbers. Then V is a vector space. We can write out the definition of a vector space and see that the set V does satisfy that definition. That's the first vector space we get to consider.
But we encounter lots more vector spaces; e.g., in 3 dimensions, a 2 dimensional plane through the origin is also a vector space.
Gee, I mentioned dimension; we need a good definition and a lot of associated theorems. Linear algebra has those.
So, for matrix A, vector x, and vector of zeros 0, the set of all solutions x to
Ax = 0
is a vector space, and it and its dimension are central in what we get in many applications, e.g., at the end of Gauss elimination, fitting linear equations to data, etc.
(2.5) Eigen Values, Vectors
Eigen in German translates to English as special, unique, singular, or some such.
Well, for a n x n matrix A, we might have that
Ax = lx
for number l. In this case what matrix A does to vector x is just change its length by l and keep its direction the same. So, l and x are quite special. Then l is an eigenvalue of A, and x is a corresponding eigenvector of A.
These eigen quantities are central to the crucial singular value decomposition, the polar decomposition, principal components, etc.
(2.6) Texts
A good, now quite old, intermediate text in linear algebra is by Hoffman and Kunze, IIRC now available for free as PDF on the Internet.
A special, advanced linear algebra text is P. Halmos, Finite Dimensional Vector Spaces written in 1942 when Halmos was an assistant to John von Neumann at the Institute for Advanced Study. The text is an elegant finite dimensional introduction to infinite dimensional Hilbert space.
At
http://www.american.com/archive/2008/march-april-magazine-co...
is an entertaining article about Harvard's course Math 55. At one time that course used that book by Halmos and also, see below, Baby Rudin.
For more there is
Richard Bellman, Introduction to Matrix Analysis.
Horn and Johnson, Matrix Analysis.
There is much more, e.g., on numerical methods. There a good start is LINPACK, the software, associated documentation, and references.
(5) More
The next two topics would be probability theory and statistics.
For a first text in either of these two, I'd suggest you find several leading research universities, call their math departments, and find what texts they are using for their first courses in probability and statistics. I'd suggest you get the three most recommended texts, carefully study the most recommended one, and use the other two for reference.
Similarly for calculus and linear algebra.
For more, that would take us into a ugrad math major. Again, make some phone calls for a list of recommended texts. One of those might be
W. Rudin, Principles of Mathematical Analysis.
aka, "Baby Rudin". It's highly precise and challenging.
For more,
H. Royden, Real Analysis
W. Rudin, Real and Complex Analysis
L. Breiman, Probability
M. Loeve, Probability
J. Neveu, Mathematical Foundations of the Calculus of Probability
The last two are challenging.
For Bayesian, that's conditional expectation from the Radon-Nikodym theorem with a nice proof by John von Neumann in Rudin's Real and Complex Analysis.
After those texts, often can derive the main results of statistics on your own or just use Wikipedia a little. E.g., for the Neyman-Pearson result in statistical hypothesis testing, there is a nice proof from the Hahn decomposition from the Radon-Nikodym theorem.
Related
A Course in Probability Theory
3 recommend · 11
Brownian motion and stochastic calculus
3 recommend · 10
Probability
5 recommend · 11
Convergence of Probability Measures
2 recommend · 14
Mathematical basis of statistics
1 recommend · 3
Measure theory
2 recommend · 10
