Applied regression analysis

What Hacker News says

1. graycatJul 2018
   Part I

   A relatively good, introductory text on statistics, starting without either calculus or linear algebra, with nice progress into experimental design and analysis of variance, that is, multivariate statistics with discrete data, from some serious experts in that field, from University of Iowa, for some serious reasons, how to maximize corn yields considering soil chemistry, water, seed variety, plowing techniques, fertilizer, etc., long commonly used for undergraduates in the social sciences, e.g., educational statistics, agriculture, etc. is (with my mark-up for TeX):

   George W.\ Snedecor and William G.\ Cochran, {\it Statistical Methods, Sixth Edition.\/}

   A good, first, no toy, text on regression, with minimal prerequisites:

   N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis.\/}

   Some good books on regression and its usual generalizations, e.g., IIRC, factor analysis, i.e., principle components, discriminate analysis, etc.:

   Maurice M.\ Tatsuoka, {\it Multivariate Analysis: Techniques for Educational and Psychological Research.\/}

   Donald F.\ Morrison, {\it Multivariate Statistical Methods: Second Edition.\/}

   William W.\ Cooley and Paul R.\ Lohnes, {\it Multivariate Data Analysis.\/}

   A mathematically relatively serious text on regression:

   C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition.\/}

   For multivariate statistics with discrete data, consider

   George W.\ Snedecor and William G.\ Cochran, {\it Statistical Methods, Sixth Edition.\/}

   Stephen E.\ Fienberg, {\it The Analysis of Cross-Classified Data.\/}

   Yvonne M.\ M.\ Bishop, Stephen E.\ Fienberg, Paul W.\ Holland, {\it Discrete Multivariate Analysis:\ \ Theory and Practice.\/}

   Shelby J.\ Haberman, {\it Analysis of Qualitative Data, Volume 1, Introductory Topics.\/}

   Shelby J.\ Haberman, {\it Analysis of Qualitative Data, Volume 2, New Developments.\/}

   The classic on the math of analysis of variance:

   Henry Scheff\'e, {\it Analysis of Variance.\/}

   Broadly, in all of this, we are trying to analyze data on, call it, several variables, make predictions, etc.

   In all of this, and as in the title

   {\it The Analysis of Cross-Classified Data.\/}

   above, suppose we have in mind random variables Y and X.

   What is a random variable? Go outside. Measure something. Call that the value of random variable Y. What you measured was one value of possibly many that you might have measured. Considering all those possible values, there is a cumulative distribution F_Y(y) that for any real number y we have the probability that random variable Y is <= real number y

   P(Y <= y) = F_Y(y).

   So, F_Y(y) is defined for all real numbers y, is at 0 at the limit of y at minus infinity and at 1 at the limit of y at plus infinity. So, as we move real number y from left to right, F_Y(y) increases -- monotonically. On a nice day, function F_Y is differentiable, and with the derivative from calculus

   f_Y(y) = d/dy F_Y(y)

   and is the probability density of real random variable Y.

   Here's the standard way to discover something about Y, in particular about its cumulative distribution F_Y:

   We can imagine having random variables Y_1, Y_2, ... that are, in the sense of probability, independent of Y and that have the same cumulative distribution as Y. Then for positive integer n, and for real number y, by the law of large numbers (the weak version has an easy proof), in the limit as n grows large, as accurately as we please, the fraction of the values

   Y_1, Y_2, ...,Y_n

   that are <= y

   is F_Y(y). So, via such simple random sampling, for any real number y we can estimate F_Y(y), the cumulative distribution of Y.

   For a little more, under meager assumptions that hold nearly universally in practice, if we take the ordinary grade school average of

   Y_1, Y_2, ..., Y_n

   as n increases we will approximate the average or expected value of Y denoted by E[Y].

   To define the expected value, we can use some calculus and the cumulative distribution of F_Y, but for now let's just use our intuition about averages and move along.

   Now suppose we are also given random variable X. Maybe the values of X are just real numbers, some 10 real numbers, 20 values, from set {1, 2, 3}, the last three weeks 100 times a second of the NYSE price of Microsoft, or full details on the atmosphere of earth every microsecond for the past 5 billion years. That is, for the values of X we can accept a lot of generality. Still more generality is possible, but that would take us on a detour for a while.

   For our point here, let's assume that X takes on just discrete values or we have just rounded off the values and forced them to be discrete. In practice we will have only finitely many discrete values.

   Now we want to use X to predict Y.

   So, sure, much of machine learning is to construct a model, maybe with regression trees or neural networks, to make this prediction, but here we will show a simpler way that is always the most accurate possible whenever we have enough data. How 'bout that!

   This simpler way is just old cross tabulation.

   Net, over a wide range of real cases trying to predict Y from X, we should just use cross tabulation unless we don't have enough data. Or, the main reason for just empirical curve fitting using regression linear models or neural network continuous models is that we don't have enough data just to use cross tabulation.

   For a preview of a coming attraction, will notice in nearly all of regression and neural networks big concerns about over fitting. Well, cross tabulation doesn't have that problem. How 'bout that!

   Show morethread
2. graycatDec 2017
   One very old tool for such things was called "stepwise regression". IIRC J. Tukey was partially involved in that. It appears that the AI/ML work is close to the regression and curve fitting going back strongly to the early days of computers in the 1960s and a lot in the social sciences back to the 1940s and even about 1900.

   A lot is known. E.g., there's the now classic Draper and Smith, Applied Regression Analysis. Software IBM Scientific Subroutine Package (SSP), SPSS (Statistical Package for the Social Sciences), SAS (Statistical Analysis System), etc. does the arithmetic for texts such as Draper and Smith. For some decades some of the best users of such applied math were the empirical macro economic model builders. E.g., once at a hearing in Congress I heard a guy, IIRC, Adams talking about that.

   Lesson: If are going to do curve fitting for model building, then a lot is known. Maybe what is new is working with millions of independent variables and trillions of bytes of data. But it stands to reason that there will also be problems with 1, 2, 1 dozen, 2 dozen variables and some thousands or millions of bytes of data, and people have been doing a lot of work like that for over half a century. Sometimes they did good work. If want to do model building on that more modest and common scale, my guess is that should look mostly at the old very well done work. Here is just a really short sampling of some of that old work:

   Stephen E. Fienberg, The Analysis of Cross-Classified Data, ISBN 0-262-06063-9, MIT Press, Cambridge, Massachusetts, 1979.

   Yvonne M. M. Bishop, Stephen E. Fienberg, Paul W. Holland, Discrete Multivariate Analysis: Theory and Practice, ISBN 0-262-52040-0, MIT Press, Cambridge, Massachusetts, 1979.

   Shelby J. Haberman, Analysis of Qualitative Data, Volume 1, Introductory Topics, ISBN 0-12-312501-4, Academic-Press, 1978.

   Shelby J. Haberman, Analysis of Qualitative Data, Volume 2, New Developments, ISBN 0-12-312502-2, Academic-Press, 1979.

   Henry Scheffe, Analysis of Variance, John Wiley and Sons, New York, 1967.

   C. Radhakrishna Rao, Linear Statistical Inference and Its Applications: Second Edition, ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.

   N. R. Draper and H. Smith, Applied Regression Analysis, John Wiley and Sons, New York, 1968.

   Leo Breiman, Jerome H. Friedman, Richard A. Olshen, Charles J. Stone, Classification and Regression Trees, ISBN 0-534-98054-6, Wadsworth & Brooks/Cole, Pacific Grove, California, 1984.

   There is a lesson about curve fitting: There was the ancient Greek Ptolemy who took data on the motions of the planets and fitted circles and circles inside circles, etc. and supposedly, except for some use of Kelly's Variable Constant and Finkel's Fudge Factor, got good fits. The problem, his circles had next to nothing to do with planetary motion; instead, that's based on ellipses and that was from more observations, Kepler, and Newton. Lesson: Empirical curve fitting is not the only approach.

   Actually the more mathematical statistics texts, e.g, the ones with theorems and proofs, say, "We KNOW that our system is linear and has just these variables and we KNOW about the statistical properties of our data, e.g., Gaussian errors, independent and identically distributed, and ALL we want to do is just get some good estimates of the coefficients with confidence intervals and t-tests and confidence intervals on predicted values. Then, can go through all that statistics and see how to do that. But notice the assumptions at the beginning: We KNOW the system is linear, etc. and are ONLY trying to estimate the coefficients that we KNOW exist. That's long been a bit distant from practice and is apparently still farther from current ML practice.

   Okay, ML for image processing. Okay. I am unsure about how much image processing there is to do where there is enough good data for the ML techniques to do well.

   Generally there is much, much more to what can be done with applied math, applied probability, and statistics than curve fitting. My view is that the real opportunities are in this much larger area and not in the recent comparatively small area of ML.

   E.g., my startup has some original work in applied probability. Some of that work does some things some people in statistics said could not be done. No, it's doable: But it's not in the books. What is in the books is asking too much from my data. So, the books are trying for too much, and with my data that's impossible. But I'm asking for less than is in the books, and that is possible and from my data. I can't go into details in public, but my lesson is this:

   There a lot in applied math and applications that is really powerful and not currently popular, canned, etc.

   Show morethread
3. graycatJun 2015
   If you want to go by names and titles, one of my Ph.D. dissertation advisors was J. Cohon. Since you are well acquainted with CMU ...!

   But I'm judging Moore's materials based on the materials, not his employment history.

   I have nothing against Moore; it's not about Moore or me. Instead, it's about what Moore wrote.

   Google, CMU CS aside, sorry to tell you, or maybe it's good news, think of the good news, instead of that A. Moore material, there is much, much higher quality material going way back, e.g., already by, say, 1970. There's G. Dantzig, R. Gomory, R. Bellman, G. Nemhauser, Ford and Fulkerson, P. Wolfe (e.g., Wolfe dual in quadratic programming), R. Bixby, H. Kuhn, A. Tucker (prof of the prof that was the Chair of my Graduate Board orals), D. Bertsekas, J. von Neumann, J. Nash, R. Rockafellar, W. Cunningham, and many, many more. None of these people is in computer science.

   E.g., there are stacks of books on multivariate statistics with linear discriminate analysis; there's log-linear for categorical data analysis; there's controlled Markov processes and continuous time stochastic optimal control, with, say, measurable selection, scenario aggregation. etc.; there's lots of material on resampling, the bootstrap (I have published a paper in essentially that topic); there's sufficient statistics from the Radon-Nikodym theorem; and much more.

   Okay, just in regression and multi-variate statistics, just from my bookshelf, there's:

   William W. Cooley and Paul R. Lohnes, 'Multivariate Data Analysis', John Wiley and Sons, New York, 1971.

   Maurice M. Tatsuoka, 'Multivariate Analysis: Techniques for Educational and Psychological Research', John Wiley and Sons, 1971.

   C. Radhakrishna Rao, 'Linear Statistical Inference and Its Applications: Second Edition', ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.

   N. R. Draper and H. Smith, 'Applied Regression Analysis', John Wiley and Sons, New York, 1968.

   Leo Breiman, Jerome H. Friedman, Richard A. Olshen, Charles J. Stone, 'Classification and Regression Trees', ISBN 0-534-98054-6, Wadsworth & Brooks/Cole, Pacific Grove, California, 1984.

   And their mathematical notation is quite precise.

   In simple terms, what Moore is doing in those notes is mostly, not all, some very fine, old wine, corrupted, and in new bottles with new labels. E.g., the 22 pages on game theory without mentioning just the simple linear programming solution is gentleman D- work.

   Readers would be seriously mislead and ill-served not to hear that the Moore material is inferior, really, not good. We're talking grade C, gentleman B-.

   Think of the good news: There's much, much better material long on the shelves of the research libraries although rarely as computer science.

   That's just the way it is. People here on HN should be aware of that situation.

   Show morethread
4. graycatDec 2013
   Two of the biggest 'branches' of mathematics are analysis and algebra. Essentially the first course in 'analysis' is calculus.

   As in Simmons (in this list), the two most important pillars of analysis are continuity and linearity.

   The first place to learn linearity well is 'linear algebra' -- or, much the same thing, matrix theory or finite dimensional vector spaces.

   Then the algebraic aspects can be emphasized and can lead to applications in, say, the theory of error correcting codes.

   For the analysis aspects, can continue with Hilbert and Banach spaces and more and also various applications.

   Much of the mathematics in applied mathematics, science, engineering, and technology is linear algebra.

   Here is a list of books relevant to linear algebra and/or to the course I took from Roger Horn. This list also contains maybe 60% of what I'd worked from before Horn's course that let me do well in that course.

   ========================================

   Linear Algebra

   ----------------------------------------

   Horn's course was close to

   Roger A. Horn, Charles R. Johnson, 'Matrix Analysis', 0-521-38632-2, Cambridge University Press, 1990.

   with also a few topics from

   Roger A. Horn, Charles R. Johnson, 'Topics in Matrix Analysis', 0-521-46713-6, Cambridge University Press, 1994.

   ----------------------------------------

   The course also used for reference and some topics and exercises

   Richard Bellman, 'Introduction to Matrix Analysis: Second Edition', McGraw-Hill, New York, 1970.

   This book is just packed with little results; at some point can get the impression that the author went on and on ... writing. Bellman was a very bright guy in mathematics, engineering, and medicine.

   ----------------------------------------

   Relatively easy to read and relatively close to applications, and another book Horn's course used as a reference, is

   Ben Noble, 'Applied Linear Algebra', Prentice-Hall, Englewood Cliffs, NJ, 1969.

   Some edition of this book may be a good place to start for a student interested in applications now.

   ----------------------------------------

   About the easiest reading in this list, and my first text on linear algebra, was

   D. C. Murdoch, 'Linear Algebra for Undergraduates', John Wiley and Sons, New York, 1957.

   This book is awfully old, but what it has are still the basics.

   ----------------------------------------

   For my undergraduate honors paper I made some use of, and later read carefully nearly all of,

   Evar D. Nering, 'Linear Algebra and Matrix Theory', John Wiley and Sons, New York, 1964.

   The main part of this book is a relatively solid start, maybe a bit terse and advanced for a first text. The book also has in the back a collection of advanced topics, some of which might be quite good to know at some point and difficult to get elsewhere.

   One of the topics in the back is linear programming, and for that I'd recommend something else, e.g., Chv'atal and/or Bazaraa and Jarvis in this list.

   ----------------------------------------

   Likely the crown jewel of books on linear algebra is

   Paul R. Halmos, 'Finite-Dimensional Vector Spaces, Second Edition', D. Van Nostrand Company, Inc., Princeton, New Jersey, 1958.

   Halmos wrote this in about 1942 when he was an assistant to von Neumann at the Institute for Advanced Study. The book is intended to be a finite dimensional introduction to Hilbert space theory, or how to do linear algebra using mostly only what also works in Hilbert space. It's likely fair to credit von Neumann with Hilbert space.

   The book is elegant.

   Apparently at one time Harvard's course Math 55, with a colorful description at,

   http://www.american.com/archive/2008/march-april-magazine-co...

   used this text by Halmos and also, as also in this list, Rudin's 'Principles' and Spivak.

   ----------------------------------------

   Long highly regarded as a linear algebra text is

   Hoffman and Kunze, 'Linear Algebra, Second Edition', Prentice-Hall, Englewood Cliffs, New Jersey, 1971.

   ========================================

   Numerical Methods

   ----------------------------------------

   If want to take numerical computations in linear algebra seriously, then consider the next book or something better if can find it

   George E. Forsythe and Cleve B. Moler, 'Computer Solution of Linear Algebraic Systems', Prentice-Hall, Englewood Cliffs, 1967.

   ========================================

   Multivariate Statistics

   My main start with multivariate statistics was

   N. R. Draper and H. Smith, 'Applied Regression Analysis', John Wiley and Sons, New York, 1968.

   Apparently later editions of this book remain of interest.

   ----------------------------------------

   A relatively serious book on 'regression analysis' is

   C. Radhakrishna Rao, 'Linear Statistical Inference and Its Applications: Second Edition', ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.

   ----------------------------------------

   Three famous, general books on multivariate statistics are

   Maurice M. Tatsuoka, 'Multivariate Analysis: Techniques for Educational and Psychological Research', John Wiley and Sons, 1971.

   William W. Cooley and Paul R. Lohnes, 'Multivariate Data Analysis', John Wiley and Sons, New York, 1971.

   Donald F. Morrison, 'Multivariate Statistical Methods: Second Edition', ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.

   ========================================

   Analysis of Variance

   A highly regarded first book on analsis of variance and experimental design is

   George W. Snedecor and William G. Cochran, 'Statistical Methods, Sixth Edition', ISBN 0-8138-1560-6, The Iowa State University Press, Ames, Iowa, 1971.

   and a famous, more mathematical, book is

   Henry Scheff'e, 'Analysis of Variance', John Wiley and Sons, New York, 1967.

   ========================================

   Linear Optimization

   A highly polished book on linear programming is

   Vav sek Chv'atal, 'Linear Programming', ISBN 0-7167-1587-2, W. H. Freeman, New York, 1983.

   ----------------------------------------

   Nicely written and with more emphasis on the important special case of network flows is

   Mokhtar S. Bazaraa and John J. Jarvis, 'Linear Programming and Network Flows', ISBN 0-471-06015-1, John Wiley and Sons, New York, 1977.

   ----------------------------------------

   A grand applied mathematics dessert buffet, based on Banach space and the Hahn-Banach theorem is

   David G. Luenberger, 'Optimization by Vector Space Methods', John Wiley and Sons, Inc., New York, 1969.

   ========================================

   Mathematical Analysis Relevant to Understanding Linearity

   ----------------------------------------

   Long the first place a math student gets a fully serious encounter with calculus and closely related topics has been

   Walter Rudin, 'Principles of Mathematical Analysis, Third Edition', McGraw-Hill, New York, 1964.

   The first chapters of this book do well as an introduction to metric spaces, and that work applies fully to vector spaces.

   ----------------------------------------

   A nice place to get comfortable doing mathematics in several dimensions is

   Wendell H. Fleming, 'Functions of Several Variables', Addison-Wesley, Reading, Massachusetts, 1965.

   Some of the material here is also good for optimization.

   ----------------------------------------

   Another place to get comfortable doing mathematics in several dimensions is

   Michael Spivak, {\it Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus,\/} W.\ A.\ Benjamin, New York, 1965.\ \

   ----------------------------------------

   The first half, the 'real' half of the next book has polished introductions to Hilbert and Banach spaces which are some of the most important vector spaces

   Walter Rudin, 'Real and Complex Analysis', ISBN 07-054232-5, McGraw-Hill, New York, 1966.

   ----------------------------------------

   An elegant introduction to how to get comfortable in metric space is

   George F. Simmons, 'Introduction to Topology and Modern Analysis', McGraw Hill, New York, 1963.

   ========================================

   Ordinary Differential Equations

   ----------------------------------------

   Linear algebra is important, as some points crucial, for ordinary differential equations, a polished introduction from a world expert is

   Earl A. Coddington, 'An Introduction to Ordinary Differential Equations', Prentice-Hall, Englewood Cliffs, NJ, 1961.

   Show morethread
5. waldrewsMay 2008
   I agree with Feller as the classical book on probability; it is beautifully written and full of insight and deserves to be read over and over again.

   For theoretical statistics, justifying why you'd use one method versus another, the graduate level classic is Cassela and Berger, Statistical Inference. This teaches you to design your own tests and compare Bayesian vs. frequentist methods and interpret the paradoxes - and is full of beautiful explanations - but it's not a bag of tricks.

   Applied statistics book tend to be more specialized. There's plenty of cookbooks with statements like "and then you should do this regression and compare the p-value to 5%," but that's not what you want. Any of several books with titles like Applied Regression Analysis or Multivariate Statistical Analysis will be at a more substantial level. After that there are specialist topics - experiment design, survival analysis, heavy tailed distributions, robust statistics, multi-level modeling, asymptotics. Statisticians get involved in a lot of areas, from quality control in manufacturing to clinical trial design to economic forecasting to genetics, so there's a lot of overlap with other fields in the academic literature.

   Econometric Analysis by Greene is the economists' favorite stats book, and is very self-contained, covering everything from computational tricks for Monte Carlo simulations top some very modern multivariate methods.

   As a computer scientist/hacker, your natural starting point is Bishop. Pattern recognition and Machine Learning or perhaps http://www.ai.mit.edu/courses/6.867-f03/lectures.html

   I've heard machine learning defined as the application of statistical methods to engineering; people with statistics degrees grumble that CS/machine learning types learn to do what's computationally feasible and not properly justify their methods, doing the sort of analysis you'd see in Casella and Berger.

   In terms of software, a lot of statistical computing in the open source world now gets done on R platform. For modern computationally intensive methods like MCMC (Markov Chain Monte Carlo) the program BUGS/WinBUGS is a standard.

   Also, http://videolectures.net/ has a whole lot of lectures (mostly from the Machine Learning perspective), starting with intro to probability and going right up to modern research.

   Show morethread
6. graycatMar 2018
   In the course, in lecture

   "Reducing Loss: Gradient Descent"

   is

   "Convex problems have only one minimum; that is, only one place where the slope is exactly 0. That minimum is where the loss function converges."

   The first sentence is flatly wrong: E.g., for positive integer n and the set of real numbers R, function f: R^n --> R where for all x in R^n f(x) = 0, f is convex, concave, and linear, and for all x in R^n x is a minimum and a maximum of f.

   Can there be uncountably infinitely many alternative minima for the Google ML problems? Yes, e.g., just enter one of the independent variables twice.

   The second sentence is nonsense.

   Grotesque, outrageous incompetence!!!!

   It has long been known that minimizing a convex function, even a differentiable convex function, with just gradient descent can be just horribly inefficient. A LOT is known about how to do much better than just gradient descent. E.g., there is Newton iteration (right, that Newton, hundreds of years ago) and quasi-Newton. And there's more.

   Why so inefficient? Well, draw a picture like Google did except use just two independent variables instead of just the one in the Google picture. Then see that the resulting, convex "bowl" can be like a long, narrow boat with a very gentle slope in one direction and a very steep slope in an orthogonal direction. Yes the cross section of the bowl can be, first cut, an ellipse with one short axis and one long one. Sure, the axes are eigenvectors, etc. and the ellipse is part of a local quadratic approximation. Well, gradient descent keeps going back and forth nearly parallel to the short axis of the ellipse and making nearly no progress on the long axis. People have known this and known good things to do about it for, uh, at least half a century.

   For the Google ML problems, might (1) tweak Newton iteration to improve the rate of convergence to a minimum or (2) at each iteration don't get a gradient descent but get a supporting hyperplane of the epigraph of the convex function, as the iterations proceed, accumulate these hyperplanes, notice that they lead to an approximation of the full epigraph, and use linear programming or some tweak of that to minimize the hyperplane approximation to the convex function -- there is much more that can be said here. By the way, the convex function for that ML problem is quite special, e.g., quadratic.

   A lot has long been known and well polished in regression and classification, e.g., with TeX markup:

   N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \

   Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \

   C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \

   A good start on convexity is

   Wendell H.\ Fleming, {\it Functions of Several Variables,\/} Addison-Wesley, Reading, Massachusetts, 1965.\ \

   Total, fun, dessert ice cream on convexity is Jensen's inequality; right away can use it to prove a lot of classic inequalities.

   Gee, look on the upside!!!! From this sample, the claims of machine learning (ML) revolutionizing the economy are nonsense!!!! And for startups, don't much have to worry about serious competition from Google!!!!

   Show morethread