The Signal and the Noise
Nate Silver built an innovative system for predicting baseball performance, predicted the 2008 election within a hair’s breadth, and became a national sensation as a blogger—all by the time he was thirty. The New York Times now publishes FiveThirtyEight.com, where Silver is one of the nation’s most influential political forecasters. Drawing on his own groundbreaking work, Silver examines the world of prediction, investigating how we can distinguish a true signal from a universe of noisy data. Most predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. This is the “prediction paradox”: The more humility we have about our ability to make predictions, the more successful we can be in planning for the future. In keeping with his own aim to seek truth from data, Silver visits the most successful forecasters in a range of areas, from hurricanes to baseball, from the poker table to the stock market, from Capitol Hill to the NBA. He explains and evaluates how these forecasters think and what bonds they share. What lies behind their success? Are they good—or just lucky? What patterns have they unraveled? And are their forecasts really right?
Stats for nerds
47recommend
97mentions
79people
2012–2026years
57% recommend43% neutral
Mentions per year
What Hacker News says
- There are a number of good books to explain the main concepts of statistics without getting bogged down in the technical details. As a grad student at a public university in the USA, I worked as a teaching assistant for the undergraduate course Introduction to Statistics. The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered a good textbook (listed below), many of the students struggled with the main concepts, so my colleagues and I added a optional supplemental reading list to the syllabus. The following books are the ones we recommended to the students. I have read all these books, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend].
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences.
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't.
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition.
Nassim Nicholas Taleb (2010) The Black Swan: The Impact of the Highly Improbable, second edition.
- As a grad student at a public university in the USA, I worked as a teaching assistant for the sociology class "Introduction to Statistics." The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered an excellent textbook (listed below), most of the undergraduate students struggled with the material, so my colleagues and I added a supplemental reading list to the syllabus for those students interested in studying the main concepts. The following books are the ones we recommended to the students. I have read all these texts, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend]
Catherine Marsh and Jane Elliott (2008) Exploring Data: An Introduction to Data Analysis for Social Scientists, second edition
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition;
_______ (2010) The Black Swan: The Impact of the Highly Improbable, second edition
- At this point in my life I have zero patience or sympathy for the story of a man selling his company to a massive conglomerate and then feeling betrayed or somehow sad/regretful when said conglomerate destroys it or weaponizes it. I'm simply tired of this hindsight virtue signaling. They don't care about us. That means even you, Nate Silver. Btw was a big fan back then! Signal and the Noise was a great book.
- I got into audiobooks this year and so have “read” a lot more than usual. Here are some of the most memorable:
Outlive by Attia - wouldn’t take everything he says as gospel but he helped me focus on what is really important in health and fitness and why
I am a Strange Loop by Hofstadter - trying to get a grasp on Gödels incompleteness theorem, but this book is a lot more. I particularly enjoyed the bits about Albert Schweitzer, and the chapter on his late wife, how we host the souls of others crudely on our own hardware. Helpful book in the age of AI.
The Man from the Future by Bhattacharya, a Von Neumann biography. Really helped understand the context of this great man’s achievements.
On the Edge by Silver. The signal and the noise was a lot better (which I read last year)
Benjamin Franklin by Isaacson. Fascinating renaissance man, interesting tour through US history
Can’t Hurt Me by Goggins. This book helped me through some emotionally difficult times and has some really great life advice (mixed with some really self-destructive behavior)
The Misbehavior of Markets by Mandelbrot, Hudson. Currently reading this, so far pretty fascinating
- I am not the person you asked, but I have been on a similar path to improve my statistical literacy. For context, I am fairly good at math generally (used to be a math major in college decades ago; didn't do particularly well though I did graduate) but always managed to get myself extremely confused when it comes to statistics.
In terms of books: there are a few good ones aimed for the general public, such as The Signal and The Noise. How to Measure Anything: Finding the Value of Intangibles in Business is a good book of applying statistical thinking in a practical setting, though it wouldn't help you wrap your brain around things like the Monty Hall problem.
The one book that really made things click for me was this:
Probability Theory: The Logic of Science by E. T. Jaynes
This book is a bit more math-heavy, but I think anyone with a working background in a science or engineering field (including software engineering) should be able to get the important fundamental idea out of the book.
You don't need to completely comprehend all the details in math (I surely didn't); it is enough to have a high-level understanding of how the formulas are structured at the high level. But you do need enough math (for example, an intuitive understanding of logarithm) for the book to be useful.
- I second both The Signal and the Noise as well as How to Measure Anything. I also mentioned upthread Willful Ignorance.
I think perhaps the best bang for your buck could be Wheeler's Understanding Variation -- but that is based mainly on vague memory and skimming the table of contents. I plan on writing a proper review of that book in the coming year to make certain it is what I remember it to be.
I think the earlier works by Taleb also touch on this (Fooled by Randomness seems to have it in the title).
But then I strongly recommend branching out to places where these fundamentals are used, to cement them:
- Anything popular by Deming (e.g. The New Economics)
- Anything less popular by Deming (e.g. Some Theory of Sampling)
- Moneyball
- Theory of Probability (de Finetti)
- Causality (Pearl)
- Applied Survival Analysis
- Analysis of Extremal Events
- Regression Modeling with Actuarial and Financial Applications
The more theoretical and applied books are less casual reads, obviously. They also happen to be the directions in which I have gone -- you may have more luck picking your own direction for where to apply and practice your intuition.
Edit: Oh and while I don't have a specific book recommendation because none of the ones I read I have good opinions on, something on combinatorics helps with getting a grasp on the general smell of entropy and simpler problems like Monty Hall.
