Name: Linear statistical inference and its applications
ISBN: 9780471708230

What Hacker News says

graycatJul 2018

Part I

A relatively good, introductory text on statistics, starting without either calculus or linear algebra, with nice progress into experimental design and analysis of variance, that is, multivariate statistics with discrete data, from some serious experts in that field, from University of Iowa, for some serious reasons, how to maximize corn yields considering soil chemistry, water, seed variety, plowing techniques, fertilizer, etc., long commonly used for undergraduates in the social sciences, e.g., educational statistics, agriculture, etc. is (with my mark-up for TeX):

George W.\ Snedecor and William G.\ Cochran, {\it Statistical Methods, Sixth Edition.\/}

A good, first, no toy, text on regression, with minimal prerequisites:

N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis.\/}

Some good books on regression and its usual generalizations, e.g., IIRC, factor analysis, i.e., principle components, discriminate analysis, etc.:

Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research.\/}

Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition.\/}

William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis.\/}

A mathematically relatively serious text on regression:

C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition.\/}

For multivariate statistics with discrete data, consider

George W.\ Snedecor and William G.\ Cochran, {\it Statistical Methods, Sixth Edition.\/}

Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data.\/}

Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice.\/}

Shelby J.\ Haberman, {\it Analysis of Qualitative Data, Volume 1, Introductory Topics.\/}

Shelby J.\ Haberman, {\it Analysis of Qualitative Data, Volume 2, New Developments.\/}

The classic on the math of analysis of variance:

Henry Scheff\'e, {\it Analysis of Variance.\/}

Broadly, in all of this, we are trying to analyze data on, call it, several variables, make predictions, etc.

In all of this, and as in the title

{\it The Analysis of Cross-Classified Data.\/}

above, suppose we have in mind random variables Y and X.

What is a random variable? Go outside. Measure something. Call that the value of random variable Y. What you measured was one value of possibly many that you might have measured. Considering all those possible values, there is a cumulative distribution F_Y(y) that for any real number y we have the probability that random variable Y is <= real number y

P(Y <= y) = F_Y(y).

So, F_Y(y) is defined for all real numbers y, is at 0 at the limit of y at minus infinity and at 1 at the limit of y at plus infinity. So, as we move real number y from left to right, F_Y(y) increases -- monotonically. On a nice day, function F_Y is differentiable, and with the derivative from calculus

f_Y(y) = d/dy F_Y(y)

and is the probability density of real random variable Y.

Here's the standard way to discover something about Y, in particular about its cumulative distribution F_Y:

We can imagine having random variables Y_1, Y_2, ... that are, in the sense of probability, independent of Y and that have the same cumulative distribution as Y. Then for positive integer n, and for real number y, by the law of large numbers (the weak version has an easy proof), in the limit as n grows large, as accurately as we please, the fraction of the values

Y_1, Y_2, ...,Y_n

that are <= y

is F_Y(y). So, via such simple random sampling, for any real number y we can estimate F_Y(y), the cumulative distribution of Y.

For a little more, under meager assumptions that hold nearly universally in practice, if we take the ordinary grade school average of

Y_1, Y_2, ..., Y_n

as n increases we will approximate the average or expected value of Y denoted by E[Y].

To define the expected value, we can use some calculus and the cumulative distribution of F_Y, but for now let's just use our intuition about averages and move along.

Now suppose we are also given random variable X. Maybe the values of X are just real numbers, some 10 real numbers, 20 values, from set {1, 2, 3}, the last three weeks 100 times a second of the NYSE price of Microsoft, or full details on the atmosphere of earth every microsecond for the past 5 billion years. That is, for the values of X we can accept a lot of generality. Still more generality is possible, but that would take us on a detour for a while.

For our point here, let's assume that X takes on just discrete values or we have just rounded off the values and forced them to be discrete. In practice we will have only finitely many discrete values.

Now we want to use X to predict Y.

So, sure, much of machine learning is to construct a model, maybe with regression trees or neural networks, to make this prediction, but here we will show a simpler way that is always the most accurate possible whenever we have enough data. How 'bout that!

This simpler way is just old cross tabulation.

Net, over a wide range of real cases trying to predict Y from X, we should just use cross tabulation unless we don't have enough data. Or, the main reason for just empirical curve fitting using regression linear models or neural network continuous models is that we don't have enough data just to use cross tabulation.

For a preview of a coming attraction, will notice in nearly all of regression and neural networks big concerns about over fitting. Well, cross tabulation doesn't have that problem. How 'bout that!

thread

graycatDec 2013

Two of the biggest 'branches' of mathematics are analysis and algebra. Essentially the first course in 'analysis' is calculus.

As in Simmons (in this list), the two most important pillars of analysis are continuity and linearity.

The first place to learn linearity well is 'linear algebra' -- or, much the same thing, matrix theory or finite dimensional vector spaces.

Then the algebraic aspects can be emphasized and can lead to applications in, say, the theory of error correcting codes.

For the analysis aspects, can continue with Hilbert and Banach spaces and more and also various applications.

Much of the mathematics in applied mathematics, science, engineering, and technology is linear algebra.

Here is a list of books relevant to linear algebra and/or to the course I took from Roger Horn. This list also contains maybe 60% of what I'd worked from before Horn's course that let me do well in that course.

========================================

Linear Algebra

----------------------------------------

Horn's course was close to

Roger A. Horn, Charles R. Johnson, 'Matrix Analysis', 0-521-38632-2, Cambridge University Press, 1990.

with also a few topics from

Roger A. Horn, Charles R. Johnson, 'Topics in Matrix Analysis', 0-521-46713-6, Cambridge University Press, 1994.

----------------------------------------

The course also used for reference and some topics and exercises

Richard Bellman, 'Introduction to Matrix Analysis: Second Edition', McGraw-Hill, New York, 1970.

This book is just packed with little results; at some point can get the impression that the author went on and on ... writing. Bellman was a very bright guy in mathematics, engineering, and medicine.

----------------------------------------

Relatively easy to read and relatively close to applications, and another book Horn's course used as a reference, is

Ben Noble, 'Applied Linear Algebra', Prentice-Hall, Englewood Cliffs, NJ, 1969.

Some edition of this book may be a good place to start for a student interested in applications now.

----------------------------------------

About the easiest reading in this list, and my first text on linear algebra, was

D. C. Murdoch, 'Linear Algebra for Undergraduates', John Wiley and Sons, New York, 1957.

This book is awfully old, but what it has are still the basics.

----------------------------------------

For my undergraduate honors paper I made some use of, and later read carefully nearly all of,

Evar D. Nering, 'Linear Algebra and Matrix Theory', John Wiley and Sons, New York, 1964.

The main part of this book is a relatively solid start, maybe a bit terse and advanced for a first text. The book also has in the back a collection of advanced topics, some of which might be quite good to know at some point and difficult to get elsewhere.

One of the topics in the back is linear programming, and for that I'd recommend something else, e.g., Chv'atal and/or Bazaraa and Jarvis in this list.

----------------------------------------

Likely the crown jewel of books on linear algebra is

Paul R. Halmos, 'Finite-Dimensional Vector Spaces, Second Edition', D. Van Nostrand Company, Inc., Princeton, New Jersey, 1958.

Halmos wrote this in about 1942 when he was an assistant to von Neumann at the Institute for Advanced Study. The book is intended to be a finite dimensional introduction to Hilbert space theory, or how to do linear algebra using mostly only what also works in Hilbert space. It's likely fair to credit von Neumann with Hilbert space.

The book is elegant.

Apparently at one time Harvard's course Math 55, with a colorful description at,

http://www.american.com/archive/2008/march-april-magazine-co...

used this text by Halmos and also, as also in this list, Rudin's 'Principles' and Spivak.

----------------------------------------

Long highly regarded as a linear algebra text is

Hoffman and Kunze, 'Linear Algebra, Second Edition', Prentice-Hall, Englewood Cliffs, New Jersey, 1971.

========================================

Numerical Methods

----------------------------------------

If want to take numerical computations in linear algebra seriously, then consider the next book or something better if can find it

George E. Forsythe and Cleve B. Moler, 'Computer Solution of Linear Algebraic Systems', Prentice-Hall, Englewood Cliffs, 1967.

========================================

Multivariate Statistics

My main start with multivariate statistics was

N. R. Draper and H. Smith, 'Applied Regression Analysis', John Wiley and Sons, New York, 1968.

Apparently later editions of this book remain of interest.

----------------------------------------

A relatively serious book on 'regression analysis' is

C. Radhakrishna Rao, 'Linear Statistical Inference and Its Applications: Second Edition', ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.

----------------------------------------

Three famous, general books on multivariate statistics are

Maurice M. Tatsuoka, 'Multivariate Analysis: Techniques for Educational and Psychological Research', John Wiley and Sons, 1971.

William W. Cooley and Paul R. Lohnes, 'Multivariate Data Analysis', John Wiley and Sons, New York, 1971.

Donald F. Morrison, 'Multivariate Statistical Methods: Second Edition', ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.

========================================

Analysis of Variance

A highly regarded first book on analsis of variance and experimental design is

George W. Snedecor and William G. Cochran, 'Statistical Methods, Sixth Edition', ISBN 0-8138-1560-6, The Iowa State University Press, Ames, Iowa, 1971.

and a famous, more mathematical, book is

Henry Scheff'e, 'Analysis of Variance', John Wiley and Sons, New York, 1967.

========================================

Linear Optimization

A highly polished book on linear programming is

Vav sek Chv'atal, 'Linear Programming', ISBN 0-7167-1587-2, W. H. Freeman, New York, 1983.

----------------------------------------

Nicely written and with more emphasis on the important special case of network flows is

Mokhtar S. Bazaraa and John J. Jarvis, 'Linear Programming and Network Flows', ISBN 0-471-06015-1, John Wiley and Sons, New York, 1977.

----------------------------------------

A grand applied mathematics dessert buffet, based on Banach space and the Hahn-Banach theorem is

David G. Luenberger, 'Optimization by Vector Space Methods', John Wiley and Sons, Inc., New York, 1969.

========================================

Mathematical Analysis Relevant to Understanding Linearity

----------------------------------------

Long the first place a math student gets a fully serious encounter with calculus and closely related topics has been

Walter Rudin, 'Principles of Mathematical Analysis, Third Edition', McGraw-Hill, New York, 1964.

The first chapters of this book do well as an introduction to metric spaces, and that work applies fully to vector spaces.

----------------------------------------

A nice place to get comfortable doing mathematics in several dimensions is

Wendell H. Fleming, 'Functions of Several Variables', Addison-Wesley, Reading, Massachusetts, 1965.

Some of the material here is also good for optimization.

----------------------------------------

Another place to get comfortable doing mathematics in several dimensions is

Michael Spivak, {\it Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus,\/} W.\ A.\ Benjamin, New York, 1965.\ \

----------------------------------------

The first half, the 'real' half of the next book has polished introductions to Hilbert and Banach spaces which are some of the most important vector spaces

Walter Rudin, 'Real and Complex Analysis', ISBN 07-054232-5, McGraw-Hill, New York, 1966.

----------------------------------------

An elegant introduction to how to get comfortable in metric space is

George F. Simmons, 'Introduction to Topology and Modern Analysis', McGraw Hill, New York, 1963.

========================================

Ordinary Differential Equations

----------------------------------------

Linear algebra is important, as some points crucial, for ordinary differential equations, a polished introduction from a world expert is

Earl A. Coddington, 'An Introduction to Ordinary Differential Equations', Prentice-Hall, Englewood Cliffs, NJ, 1961.

thread

jruohonenOct 2023

Still remember his Linear Statistical Inference and Its Applications from 1973 (as well as the Cramer-Rao bound, which I think he put forward already in the 1940s).

thread

graycatMar 2019

By "total least squares" you mean to pick regression coefficients that minimize the squared errors between the observed values of the dependent variable and the predicted value from the regression? The predicted values are from a perpendicular projection, and we do get the Pythagorean theorem with

total sum of squares = regression sum of squares + error sum of squares

So, we minimize the error sum of squares. Does that really control on measurement error? Not in any simple way I can see.

Or, if we have errors in the measurements of the independent variables, then we are facing one of the facts of life, we don't have the error-free, true values. Not good but usually not as bad as having your marriage fail or losing your pet dog or cat! Like the video clip of a Heifetz master class where a student tried to play the D flat minor scale and at the end Heifetz assured the student that they were still alive!

Maybe what you are saying is that with enough mathematical assumptions, e.g., the famous heteroscedasticity with independent, identically distributed mean zero Gaussian errors, as the number of observations goes to infinity the errors wash out much as in the weak/strong laws of large numbers and we get to f'get about the errors -- maybe there is such a theorem, I should get out my copy of the old

C. Radhakrishna Rao, Linear Statistical Inference and Its Applications: Second Edition, ISBN 0-471-70823-2, John Wiley and Sons, New York.

and look or do some such derivations for myself.

But, to what end if we don't believe the mathematical assumptions for the mathematical theorems, e.g., heteroscedasticity with independent, identically distributed mean zero Gaussian errors?

Sorry, from 50,000 feet up, it seems to me that having control variables in regression is shaky stuff. And without some careful derivations, we should not be surprised at the effects of various errors.

Also, the usual derivations of the math are in the context of just some one regression model where we make all those assumptions. Instead, given one dependent variable and 10 independent variables plus five more we believe are causes plus 10 more we want to use as controls, the dependent variable plus 25 more variables in all, last time I checked we were short on how to pick the 2^25 sets of independent variables and make sense of the different, maybe wildly different, coefficients we get.

Here's a simple view: If we have 5 independent variables and they are all orthogonal, then we can get the regression coefficients one at a time just from 5 projections, covariances, inner products (all essentially the same things except how we scale things) and have those coefficient just the same for any of the 2^5 regression analyses. That is, if we have orthogonal independent variables U, V, W, X, and Y and dependent variable Z, then we can get the coefficients one at a time and be done -- have all the regression coefficients for all 2^5 regressions. Otherwise, without othogonality, we face some possibly tricky math derivations -- maybe they are in Rao's book, it's thick enough -- and are asking a bit too much from regression analysis.

Others have seen this swamp, and a current idea from the machine learning community, and going back at least to L. Breiman, is that we are not really looking for coefficients, t-tests on the coefficients, F-ratio tests on the regressions, confidence intervals on the predictions (for that might try some resampling ideas), importance of coefficients, causes, control variables, etc. but are just looking for a fit that can predict: To this end we put the data into at least two buckets, fit to one bucket and test in another one. Our main criterion is just that the model predicts well for the data we have. That is, all the data in all the buckets has all the same statistical assumptions, whatever the heck they are, and we are just fitting and then testing (confirming) on simple random samples of data all from some one big bucket. Yes, we still run into the issue of overfitting, fit well in the first bucket but flop terribly testing on the second bucket. Okay, a bit crude, uncouth, vulgar, primitive, ..., etc. but maybe useful in some cases -- apparently Breiman made it useful in some cases of medical data.

thread

graycatMar 2018

In the course, in lecture

"Reducing Loss: Gradient Descent"

"Convex problems have only one minimum; that is, only one place where the slope is exactly 0. That minimum is where the loss function converges."

The first sentence is flatly wrong: E.g., for positive integer n and the set of real numbers R, function f: R^n --> R where for all x in R^n f(x) = 0, f is convex, concave, and linear, and for all x in R^n x is a minimum and a maximum of f.

Can there be uncountably infinitely many alternative minima for the Google ML problems? Yes, e.g., just enter one of the independent variables twice.

The second sentence is nonsense.

Grotesque, outrageous incompetence!!!!

It has long been known that minimizing a convex function, even a differentiable convex function, with just gradient descent can be just horribly inefficient. A LOT is known about how to do much better than just gradient descent. E.g., there is Newton iteration (right, that Newton, hundreds of years ago) and quasi-Newton. And there's more.

Why so inefficient? Well, draw a picture like Google did except use just two independent variables instead of just the one in the Google picture. Then see that the resulting, convex "bowl" can be like a long, narrow boat with a very gentle slope in one direction and a very steep slope in an orthogonal direction. Yes the cross section of the bowl can be, first cut, an ellipse with one short axis and one long one. Sure, the axes are eigenvectors, etc. and the ellipse is part of a local quadratic approximation. Well, gradient descent keeps going back and forth nearly parallel to the short axis of the ellipse and making nearly no progress on the long axis. People have known this and known good things to do about it for, uh, at least half a century.

For the Google ML problems, might (1) tweak Newton iteration to improve the rate of convergence to a minimum or (2) at each iteration don't get a gradient descent but get a supporting hyperplane of the epigraph of the convex function, as the iterations proceed, accumulate these hyperplanes, notice that they lead to an approximation of the full epigraph, and use linear programming or some tweak of that to minimize the hyperplane approximation to the convex function -- there is much more that can be said here. By the way, the convex function for that ML problem is quite special, e.g., quadratic.

A lot has long been known and well polished in regression and classification, e.g., with TeX markup:

N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \

Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \

C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \

A good start on convexity is

Wendell H.\ Fleming, {\it Functions of Several Variables,\/} Addison-Wesley, Reading, Massachusetts, 1965.\ \

Total, fun, dessert ice cream on convexity is Jensen's inequality; right away can use it to prove a lot of classic inequalities.

Gee, look on the upside!!!! From this sample, the claims of machine learning (ML) revolutionizing the economy are nonsense!!!! And for startups, don't much have to worry about serious competition from Google!!!!

thread

graycatDec 2017

One very old tool for such things was called "stepwise regression". IIRC J. Tukey was partially involved in that. It appears that the AI/ML work is close to the regression and curve fitting going back strongly to the early days of computers in the 1960s and a lot in the social sciences back to the 1940s and even about 1900.

A lot is known. E.g., there's the now classic Draper and Smith, Applied Regression Analysis. Software IBM Scientific Subroutine Package (SSP), SPSS (Statistical Package for the Social Sciences), SAS (Statistical Analysis System), etc. does the arithmetic for texts such as Draper and Smith. For some decades some of the best users of such applied math were the empirical macro economic model builders. E.g., once at a hearing in Congress I heard a guy, IIRC, Adams talking about that.

Lesson: If are going to do curve fitting for model building, then a lot is known. Maybe what is new is working with millions of independent variables and trillions of bytes of data. But it stands to reason that there will also be problems with 1, 2, 1 dozen, 2 dozen variables and some thousands or millions of bytes of data, and people have been doing a lot of work like that for over half a century. Sometimes they did good work. If want to do model building on that more modest and common scale, my guess is that should look mostly at the old very well done work. Here is just a really short sampling of some of that old work:

Stephen E. Fienberg, The Analysis of Cross-Classified Data, ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.

Yvonne M. M. Bishop, Stephen E. Fienberg, Paul W. Holland, Discrete Multivariate Analysis: Theory and Practice, ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.

Shelby J. Haberman, Analysis of Qualitative Data, Volume 1, Introductory Topics, ISBN 0-12-312501-4, Academic-Press, 1978.

Shelby J. Haberman, Analysis of Qualitative Data, Volume 2, New Developments, ISBN 0-12-312502-2, Academic-Press, 1979.

Henry Scheffe, Analysis of Variance, John Wiley and Sons, New York, 1967.

C. Radhakrishna Rao, Linear Statistical Inference and Its Applications: Second Edition, ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.

N. R. Draper and H. Smith, Applied Regression Analysis, John Wiley and Sons, New York, 1968.

Leo Breiman, Jerome H. Friedman, Richard A. Olshen, Charles J. Stone, Classification and Regression Trees, ISBN 0-534-98054-6, Wadsworth & Brooks/Cole, Pacific Grove, California, 1984.

There is a lesson about curve fitting: There was the ancient Greek Ptolemy who took data on the motions of the planets and fitted circles and circles inside circles, etc. and supposedly, except for some use of Kelly's Variable Constant and Finkel's Fudge Factor, got good fits. The problem, his circles had next to nothing to do with planetary motion; instead, that's based on ellipses and that was from more observations, Kepler, and Newton. Lesson: Empirical curve fitting is not the only approach.

Actually the more mathematical statistics texts, e.g, the ones with theorems and proofs, say, "We KNOW that our system is linear and has just these variables and we KNOW about the statistical properties of our data, e.g., Gaussian errors, independent and identically distributed, and ALL we want to do is just get some good estimates of the coefficients with confidence intervals and t-tests and confidence intervals on predicted values. Then, can go through all that statistics and see how to do that. But notice the assumptions at the beginning: We KNOW the system is linear, etc. and are ONLY trying to estimate the coefficients that we KNOW exist. That's long been a bit distant from practice and is apparently still farther from current ML practice.

Okay, ML for image processing. Okay. I am unsure about how much image processing there is to do where there is enough good data for the ML techniques to do well.

Generally there is much, much more to what can be done with applied math, applied probability, and statistics than curve fitting. My view is that the real opportunities are in this much larger area and not in the recent comparatively small area of ML.

E.g., my startup has some original work in applied probability. Some of that work does some things some people in statistics said could not be done. No, it's doable: But it's not in the books. What is in the books is asking too much from my data. So, the books are trying for too much, and with my data that's impossible. But I'm asking for less than is in the books, and that is possible and from my data. I can't go into details in public, but my lesson is this:

There a lot in applied math and applications that is really powerful and not currently popular, canned, etc.

thread