Stats for nerds
2recommend
8mentions
3people
2010–2020years
50% recommend50% neutral
Mentions per year
What Hacker News says
- Linear algebra, and more than one such book.
IMHO long and still the best linear algebra book is
Halmos, Finite Dimensional Vector Spaces (FDVS).
It was written in 1942 when Halmos was an "assistant" to John von Neumann at the Institute for Advanced Study. It is intended to be finite dimensional vector spaces but done with the techniques of Hilbert space. The central result in the book, according to Halmos, is the spectral decomposition. One result at a time, the quality of von Neumann comes through. Commonly physicists have been given that book as their introduction to Hilbert space for quantum mechanics.
But FDVS is a little too much for a first book on linear algebra, or maybe even a second book, should be maybe a third one.
Also high quality is Nering, Linear Algebra and Matrix Theory. Again, the quality comes through: Nering was a student of Artin at Princeton. There Nering does most of linear algebra on just finite fields, not just the real and complex fields; finite fields in linear algebra are important in error correcting codes. So, that finite field work is a good introduction to abstract algebra.
For a first book on linear algebra, I'd recommend something easy. The one I used was
Murdoch, Linear Algebra for Undergraduates.
It's still okay if can find it.
For a first book, likely the one by Strang at MIT is good. Just use it as a first book and don't take it too seriously since are going to cover all of it and more again later.
I can recommend the beginning sections on vector spaces, convexity, and the inverse and implicit function theorems in
Fleming, Functions of Several Variables
Fleming was long at the Brown University Division of Applied Math. The later chapters are on measure theory, the Lebesgue integral, and the exterior algebra of differential forms, and there are better treatments.
Also there is now
Stephen Boyd and Lieven Vandenberghe, Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares
at
http://vmls-book.stanford.edu/vmls.pdf
Since the book is new, I've only looked through it -- it looks like a good selection and arrangement of topics. And Boyd is good, wrote a terrific book, maybe, IMHO likely, the best in the world, on convexity, which is in a sense is half of linearity.
Some course slides are available at
http://vmls-book.stanford.edu/
For reference for more, have a copy of
Richard Bellman, Introduction to Matrix Analysis: Second Edition.
Bellman was famous for dynamic programming.
For computations in linear algebra, consider
George E. Forsythe and Cleve B. Moler, Computer Solution of Linear Algebraic Systems
although now the Linpack materials might be a better starting point for numerical linear algebra. Numerical linear algebra is now a well developed specialized field, and the Linpack materials might be a good start on the best of the field. Such linear algebra is apparently the main yardstick in evaluating the highly parallel supercomputers.
After linear algebra go through
Rudin, Principles of Mathematical Analysis, Third Edition.
He does the Riemann integral very carefully, Fourier series, vector analysis via exterior algebra, and has the inverse and implicit function theorems (key to differential geometry, e.g., for relativity theory) as exercises.
All of this material is to get to the main goals of measure theory, the Lebesgue integral, Fourier theory, Hilbert space and Banach space as in, say, the first, real (not complex) half of
Rudin, Real and Complex Analysis
But for that I would start with
Royden, Real Analysis
sweetheart writing on that math.
Depending on the math department, those books might be enough to pass the Ph.D. qualifying exam in Analysis. It was for me: From those books I did the best in the class on that exam.
Moreover, from independent study of Halmos, Nering, Fleming, Forsythe, linearity in statistics, and some more, I totally blew away all the students in a challenging second (maybe intentionally flunk out), advanced course in linear algebra and, then, did the best in the class on the corresponding qualifying exam, that is, where that second course was my first formal course in linear algebra.
Lesson: Just self study of those books can give a really good background in linear algebra and its role in the rest of pure and applied math.
No joke, linear algebra, and the associated vector spaces, is one of the most important courses for more work in pure and applied math, engineering, and likely the future of computing.
- To make the study easier and better rounded, just add some material.
Rudin's Principles is really nice, especially in retrospect once understand it, but going in as a student it can seem quite severe. It's precise but not too severe -- he just makes you go a chapter or two before it becomes clear why he is doing what he is doing.
To help, my nutshell view is that mostly he is just trying to develop the Riemann (Stieltjes) integral. His main result is, the Riemann integral exists for continuous functions on compact sets. So, then, he needs to say what a compact set is. Well, the most relevant example is just a closed interval of real numbers such as [a,b]. So, why compact? Because every continuous function on a compact set is uniformly continuous, and that lets us know that the Riemann sums converge. What is compact? Every open cover has a finite subcover, and that lets us get uniform continuity. And, in R^n, a set is compact if and only if it is closed and bounded. So, Rudin needs to talk about closed versus open -- he does that on metric spaces although really he needs it only on R^n.
So, net, he starts with metric spaces and discusses open, closed, and compact. Then he shows that in R^n, compact is the same as closed and bounded. He shows that a continuous function on a compact set is uniformly continuous. Then, presto, he shows that the Riemann (or Riemann-Stieltjes if you wish) sums converge and the Riemann integral exists.
He does some nice work on infinite sequences and series, and the main reason is that he uses those tools to show lots of limits exist, e.g., for sines, cosines, and Fourier series.
There's more of value in Principles, but IMHO I gave you a good start to make the book easier. I wish I'd had been given that outline when I was working through Principles at 1+ hour a page.
But the Lebesgue integral in Royden is the one to take fully seriously.
Make Halmos the second or third text on linear algebra. And then look at some quantum mechanics where they discuss eigenvalues, eigenvectors, Hermitian, unitary, and the spectral decomposition! Right, the Halmos book is baby Hilbert space. Then look at some applied connections, e.g.,
George E.\ Forsythe and Cleve B.\ Moler, {\it Computer Solution of Linear Algebraic Systems,\/}
Maybe spend an evening on the documentation of LINPACK.
Some weekend of great fun, take a fast pass through the Gauss, ..., Stokes theorem parts of
Tom M.\ Apostol, {\it Mathematical Analysis: A Modern Approach to Advanced Calculus,\/}
where don't take the proofs very seriously but to see how physical science and engineering look at calculus of several variables.
Neveu is a great last probability text but not a good first text. So, before Neveu, look quickly, not very seriously, at whatever, including in some introductory statistics texts.
Also, Breiman's Probability is easier to read than Neveu. So, is K. L. Chung's competitive book. And there are others.
There is some more advanced material, e.g.,
Ioannis Karatzas and Steven E.\ Shreve, {\it Brownian Motion and Stochastic Calculus, Second Edition,\/}
Good luck!
- Two of the biggest 'branches' of mathematics are analysis and algebra. Essentially the first course in 'analysis' is calculus.
As in Simmons (in this list), the two most important pillars of analysis are continuity and linearity.
The first place to learn linearity well is 'linear algebra' -- or, much the same thing, matrix theory or finite dimensional vector spaces.
Then the algebraic aspects can be emphasized and can lead to applications in, say, the theory of error correcting codes.
For the analysis aspects, can continue with Hilbert and Banach spaces and more and also various applications.
Much of the mathematics in applied mathematics, science, engineering, and technology is linear algebra.
Here is a list of books relevant to linear algebra and/or to the course I took from Roger Horn. This list also contains maybe 60% of what I'd worked from before Horn's course that let me do well in that course.
========================================
Linear Algebra
----------------------------------------
Horn's course was close to
Roger A. Horn, Charles R. Johnson, 'Matrix Analysis', 0-521-38632-2, Cambridge University Press, 1990.
with also a few topics from
Roger A. Horn, Charles R. Johnson, 'Topics in Matrix Analysis', 0-521-46713-6, Cambridge University Press, 1994.
----------------------------------------
The course also used for reference and some topics and exercises
Richard Bellman, 'Introduction to Matrix Analysis: Second Edition', McGraw-Hill, New York, 1970.
This book is just packed with little results; at some point can get the impression that the author went on and on ... writing. Bellman was a very bright guy in mathematics, engineering, and medicine.
----------------------------------------
Relatively easy to read and relatively close to applications, and another book Horn's course used as a reference, is
Ben Noble, 'Applied Linear Algebra', Prentice-Hall, Englewood Cliffs, NJ, 1969.
Some edition of this book may be a good place to start for a student interested in applications now.
----------------------------------------
About the easiest reading in this list, and my first text on linear algebra, was
D. C. Murdoch, 'Linear Algebra for Undergraduates', John Wiley and Sons, New York, 1957.
This book is awfully old, but what it has are still the basics.
----------------------------------------
For my undergraduate honors paper I made some use of, and later read carefully nearly all of,
Evar D. Nering, 'Linear Algebra and Matrix Theory', John Wiley and Sons, New York, 1964.
The main part of this book is a relatively solid start, maybe a bit terse and advanced for a first text. The book also has in the back a collection of advanced topics, some of which might be quite good to know at some point and difficult to get elsewhere.
One of the topics in the back is linear programming, and for that I'd recommend something else, e.g., Chv'atal and/or Bazaraa and Jarvis in this list.
----------------------------------------
Likely the crown jewel of books on linear algebra is
Paul R. Halmos, 'Finite-Dimensional Vector Spaces, Second Edition', D. Van Nostrand Company, Inc., Princeton, New Jersey, 1958.
Halmos wrote this in about 1942 when he was an assistant to von Neumann at the Institute for Advanced Study. The book is intended to be a finite dimensional introduction to Hilbert space theory, or how to do linear algebra using mostly only what also works in Hilbert space. It's likely fair to credit von Neumann with Hilbert space.
The book is elegant.
Apparently at one time Harvard's course Math 55, with a colorful description at,
http://www.american.com/archive/2008/march-april-magazine-co...
used this text by Halmos and also, as also in this list, Rudin's 'Principles' and Spivak.
----------------------------------------
Long highly regarded as a linear algebra text is
Hoffman and Kunze, 'Linear Algebra, Second Edition', Prentice-Hall, Englewood Cliffs, New Jersey, 1971.
========================================
Numerical Methods
----------------------------------------
If want to take numerical computations in linear algebra seriously, then consider the next book or something better if can find it
George E. Forsythe and Cleve B. Moler, 'Computer Solution of Linear Algebraic Systems', Prentice-Hall, Englewood Cliffs, 1967.
========================================
Multivariate Statistics
My main start with multivariate statistics was
N. R. Draper and H. Smith, 'Applied Regression Analysis', John Wiley and Sons, New York, 1968.
Apparently later editions of this book remain of interest.
----------------------------------------
A relatively serious book on 'regression analysis' is
C. Radhakrishna Rao, 'Linear Statistical Inference and Its Applications: Second Edition', ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.
----------------------------------------
Three famous, general books on multivariate statistics are
Maurice M. Tatsuoka, 'Multivariate Analysis: Techniques for Educational and Psychological Research', John Wiley and Sons, 1971.
William W. Cooley and Paul R. Lohnes, 'Multivariate Data Analysis', John Wiley and Sons, New York, 1971.
Donald F. Morrison, 'Multivariate Statistical Methods: Second Edition', ISBN 0-07-043186-8, McGraw-Hill, New York, 1976.
========================================
Analysis of Variance
A highly regarded first book on analsis of variance and experimental design is
George W. Snedecor and William G. Cochran, 'Statistical Methods, Sixth Edition', ISBN 0-8138-1560-6, The Iowa State University Press, Ames, Iowa, 1971.
and a famous, more mathematical, book is
Henry Scheff'e, 'Analysis of Variance', John Wiley and Sons, New York, 1967.
========================================
Linear Optimization
A highly polished book on linear programming is
Vav sek Chv'atal, 'Linear Programming', ISBN 0-7167-1587-2, W. H. Freeman, New York, 1983.
----------------------------------------
Nicely written and with more emphasis on the important special case of network flows is
Mokhtar S. Bazaraa and John J. Jarvis, 'Linear Programming and Network Flows', ISBN 0-471-06015-1, John Wiley and Sons, New York, 1977.
----------------------------------------
A grand applied mathematics dessert buffet, based on Banach space and the Hahn-Banach theorem is
David G. Luenberger, 'Optimization by Vector Space Methods', John Wiley and Sons, Inc., New York, 1969.
========================================
Mathematical Analysis Relevant to Understanding Linearity
----------------------------------------
Long the first place a math student gets a fully serious encounter with calculus and closely related topics has been
Walter Rudin, 'Principles of Mathematical Analysis, Third Edition', McGraw-Hill, New York, 1964.
The first chapters of this book do well as an introduction to metric spaces, and that work applies fully to vector spaces.
----------------------------------------
A nice place to get comfortable doing mathematics in several dimensions is
Wendell H. Fleming, 'Functions of Several Variables', Addison-Wesley, Reading, Massachusetts, 1965.
Some of the material here is also good for optimization.
----------------------------------------
Another place to get comfortable doing mathematics in several dimensions is
Michael Spivak, {\it Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus,\/} W.\ A.\ Benjamin, New York, 1965.\ \
----------------------------------------
The first half, the 'real' half of the next book has polished introductions to Hilbert and Banach spaces which are some of the most important vector spaces
Walter Rudin, 'Real and Complex Analysis', ISBN 07-054232-5, McGraw-Hill, New York, 1966.
----------------------------------------
An elegant introduction to how to get comfortable in metric space is
George F. Simmons, 'Introduction to Topology and Modern Analysis', McGraw Hill, New York, 1963.
========================================
Ordinary Differential Equations
----------------------------------------
Linear algebra is important, as some points crucial, for ordinary differential equations, a polished introduction from a world expert is
Earl A. Coddington, 'An Introduction to Ordinary Differential Equations', Prentice-Hall, Englewood Cliffs, NJ, 1961.
- Yup, that was a day one lesson in numerical linear algebra.
Another lesson: At times will find that are forming 'inner products', that is, for some positive integer n and for, say, single precision, floating point a(i), b(i) for i = 1, 2, ..., n, computing
a(1) x b(1) + a(2) x b(2) + ,,, + a(n) x b(n)
Well, do that computation in double precision. Convert the result back to single precision if necessary; still definitely, if possible do 'double precision inner product accumulation'.
Why? The first rule of numerical analysis is to avoid the difference of two large numbers whose difference is small.
Why? Because maybe 12345.6 and 12345.7 each have 6 decimal digits of precision but they differ by 0.1 which has only 1 significant digit of precision and, thus, is not very accurate and can feed inaccuracy into the subsequent calculations.
So, when accumulating an inner product, a product a(1) x b(1) is basically in double precision, so KEEP THOSE DIGITS, especially since the additions may be the small differences of large numbers. So, by doing double precision inner product accumulation, have a chance of having full single precision accuracy for the result.
Next, when you have a solution x^1 to Ax = b, substitute and get Ax^1 = b^1, form c^1 = b - b^1 as an 'error', and then solve Ay^1 = c^1 and correct for the error by setting x^2 = x^1 + y^1. So, if y^1 is accurate, then get
Ax^2 = A(x^1 + y^1) = b^1 + c^1 = b^1 + b - b^1 = b
Of course, could do several steps of iteration here.
This process is called 'iterative improvement'.
This and more are in
George E. Forsythe and Cleve B. Moler, 'Computer Solution of Linear Algebraic Systems', Prentice-Hall, Englewood Cliffs, 1967.
E.g., pay close attention to 'partial pivoting'.
Beware of the fact that Ax = b may have a solution but not a unique solution. If this is the case in your problem, then it would be hopeless to look for A^(-1) yet a solution might still be easy to find.
Note that if there is a solution, then can (1) notice that each of the numbers is in 'floating point' and, thus, just an integer times a power of 2 (10, whatever, assume 2). Thus, in Ax = b can multiply A and b by an appropriate power of 2 and, then, have a problem where all the given data is integers. Then, for some positive integer m pick m single precision integer prime numbers and for each of these numbers, using only single precision integer arithmetic, solve the system in the integers modulo that prime where find multiplicative inverses using the Euclidean greatest common divisor algorithm. Then take the resulting m results and use the Chinese remainder theorem to construct the full multiple precision rational solutions at the end. So, have solved the system exactly using only single precision integer arithmetic. See
Morris Newman, "Solving Equations Exactly," 'Journal of Research of the National Bureau of Standards -- B. Mathematics and Mathematical 'Physics', volume 71B, number 4, October-December, 1967, pages 171-179.
- Evar D. Nering, Linear Algebra and Matrix Linear Algebra and Matrix Theory*
Kenneth Hoffmann And Ray Kunze. Linear Algebra, 2nd Edition, Prentice-Hall, Englewood Cliffs, New Jersey, 1971.
https://www.zuj.edu.jo/download/linear-algebra-2nd-edition-k...
Halmos, Finite Dimensional Vector Spaces
George E. Forsythe and Cleve B. Moler, Computer Solution of Linear Algebraic Systems
Paul R. Halmos, Naive Set Theory, Van Nostrand, Princeton, NJ, 1960.
More has been done since this book, but this book is a gorgeous introduction to axiomatic set theory. So, even people who want to dig into the latest work would do well to have this as the first book. And for people wanting to read any of the more advanced material here, knowledge of this book will be from good to have to important.
https://piazza-resources.s3.amazonaws.com/is25bi6c6o1oh/isv1...
Walter Rudin, Principles of Mathematical Analysis
The third edition is a lot better than the first two.
H. L. Royden, Real Analysis: Second Edition
Beautifully written, elegant, but maybe don't work way too hard on the exercises about upper/lower semi-continuity, and there is a better summary than Littlewood's three principles.
Bernard R. Gelbaum and John M. H. Olmsted, Counterexamples in Analysis
John C. Oxtoby, Measure and Category: A Survey of the Analogies between Topological and Measure Spaces
Walter Rudin, Real and Complex Analysis
Walter Rudin, Functional Analysis
Leo Breiman, Probability
Kai Lai Chung, A Course in Probability Theory, Second Edition
Jacques Neveu, Mathematical Foundations of the Calculus of Probability
Erhan Cinlar, Introduction to Stochastic Processes
J. L. Doob, Stochastic Processes
I. I. Gihman and A. V. Skorohod, The Theory of Stochastic Processes I, II
Donald E. Knuth, The TeX book
Donald E. Knuth, The Art of Computer Programming, Second Edition
Leo Breiman, "Statistical Modeling: The Two Cultures," Statistical Science, Vol. 16, No. 3, 199–231, 2001.
Paul R. Halmos, "The Theory of Unbiased Estimation", Annals of Mathematical Statistics, Volume 17, Number 1, pages 34-43, 1946.
Paul R. Halmos and L. J. Savage, "Application of the Radon-Nikodym Theorem to the Theory of Sufficient Statistics", The Annals of Mathematical Statistics, Volume 20, Number 2 (1949), 225-241.
https://projecteuclid.org/download/pdf_1/euclid.aoms/1177730...
- I agree, but I specifically restricted my remarks to numerical linear algebra. Double precision inner product accumulation is a standard idea in numerical linear algebra, e.g., as in
George E. Forsythe and Cleve B. Moler, Computer Solution of Linear Algebraic Systems.
People have long struggled with accuracy in numerical linear algebra. At one time at the US National Bureau of Standards, the struggles were so severe that M. Newman wrote software for numerically exact solution of systems of linear equations using only machine precision integer arithmetic. How? In the given system, he multiplied by powers of 10 to get all the decimal points to the right so that he had only whole numbers. Then for each prime number in a list of some dozens of primes, he solved the system in the finite field of the integers modulo that prime. Multiplicative inverse is from the Euclidean greatest common divisor algorithm. Then using the Chinese remainder theorem, he constructed the multiprecision integer numerators and denominators of the exact rational solution. It was nice work and somewhat popular.
Careful work on condition number estimation and exploitation has also been used. Condition number can be the basis of error estimates.
Sure, for, say, the fast Fourier transform, mostly in practice 32 bits can be fine.
Related
Linear algebra and matrix theory
1 recommend · 9
Linear programming
5 recommend · 7
Introduction to Applied Linear Algebra
7 recommend · 14
Linear Algebra Done Right
130 recommend · 237
Matrix computations
19 recommend · 26
Numerical linear algebra
26 recommend · 45
2017–2024
Linear Algebra
recommend · 11
