Name: Probability Theory
ISBN: 9780521592710

Probability Theory

Publisher Description: The standard rules of probability can be interpreted as uniquely valid principles in logic. In this book, E. T. Jaynes dispels the imaginary distinction between "probability theory" and "statistical inference", leaving a logical unity and simplicity, which provides greater technical power and flexibility in applications. This book goes beyond the conventional mathematics of probability theory, viewing the subject in a wider context. New results are discussed, along with applications of probability theory to a wide variety of problems in physics, mathematics, economics, chemistry and biology. It contains many exercises and problems, and is suitable for use as a textbook on graduate level courses involving data analysis. The material is aimed at readers who are already familiar with applied mathematics at an advanced undergraduate level or higher. The book will be of interest to scientists working in any area where inference from incomplete information is necessary. Book Description: Going beyond the conventional mathematics of probability theory, this study views the subject in a wider context. It discusses new results, along with applications of probability theory to a variety of problems. The book contains many exercises and is suitable for use as a textbook on graduate-level courses involving data analysis.

What Hacker News says

aspenmartinMar 2026

Why is it important? “Statistical fit” is what you want…not understanding this is indicative of a limited understanding of what statistics is. What do you think it means to truly understand something? I don’t get it: read probability theory by Jaynes. It doesn’t really matter if the brain does Bayesian updates but that’s what’s optimal…

thread

chuanklDec 2025

Probability Theory: The Logic of Science by ET Jaynes was the book that made statistics and probability theory finally make sense to me. Before reading this book, I was already familiar with Bayesian probability, having finished (among other books) The Signal and the Noise by Nate Silver, but Jaynes took everything to a whole new level. Probability is not an objective measure of likelihood—it is a subjective measure of incomplete information. This seems like a small shift in perspective, but it is much more profound than that, and you have to read the book to understand it.

The Goal: A Process of Ongoing Improvement by Eliyahu M Goldratt opened my eyes to process optimization. It is nominally about process optimization in a manufacturing plant, but so many of the lessons can be applied to other domains, such as software development. (And also about life in general, how to think about what matters and what doesn't.) It is a bit dated now; for example The Principles of Product Development Flow offers a much more systematic treatment of the topic. But I think The Goal remains the best intro to process and systems thinking.

thread

fishmicrowaverSep 2025

Probability Theory by Jaynes if you'd like more bayes

thread

wespiser_2018May 2025

I'm not opposed to business books. It's true that most of them are in narrative form and try to extract anecdotal lessons into broad strategy, but I've found them useful for framing my own thinking about teams, strategy, and leadership. Thinking about your work, just in a different way, or though a different lens, I believe is helpful. How helpful? Probably not as much as making more decisions yourself, but at least in my environment I'm rate limited by circumstances beyond my control!

That said, there are a couple of "good" business books, and I agree with the author on the works of Michael Porter (esp. On Competition) and ET Jaynes, Probability Theory: The Logic of Science. The later was a major influence on my life and pivotal book I read as a young scientist, and it moved me into the direction of data science!

thread

JoshColeMay 2025

> it can't "think" of something entirety conceptionally new, because it doesn't really "think".

Hierarchical optimization (fast global + slow local) is a precise, implementable notion of "thinking." Whenever I've seen this pattern implemented, humans, without being told to do so by others in some forced way, seem to converge on the use of verb think to describe the operation. I think you need to blacklist the term think and avoid using it altogether if you want to think clearly about this subject, because you are allowing confusion in your use of language to come between you and understanding the mathematical objects that are under discussion.

> It can produce "new" stuff only by combining the "old" stuff in new ways,

False premise; previously debunked. Here is a refutation for you anyway, but made more extreme. Instead of modeling the language task using a pre-training predictive dataset objective, only train on a provided reward model. Such a setup never technically shows "old" stuff to the AI, because the AI is never shown stuff explicitly. It just always generates new things and then the reward model judges how well it did. Clearly, the fact that it can do generation while knowing nothing, shows that your claim that it can never generate something new -- by definition everything would be new at this point -- is clearly false. Notice that as it continually generates new things and the judgements occur, it will learn concepts.

> But LLM's don't have an understanding of things, they are statistical models that predict what statistically is most likely following the input that you give it.

Try out Jayne's Probability Theory: The Logic Of Science. Within it the various underpinning assumptions that lead to probability theory are shown to be very reasonable and normal and obviously good. Stuff like represent plausibility with real numbers, keep rankings consistent and transitive, reduce to Boolean logic at certainty, and update so you never accept a Dutch-book sure-loss -- which together force the ordinary sum and product rules of probability. Then notice that statistics is in a certain sense just what happens when you apply the rules of probability.

> also having a understanding of certain concepts that allows you to arrive at new points like C, D, E, etc. But LLM's don't have an understanding of things

This is also false. Look into the line of research that tends to go by the name of Circuits. Its been found that models have spaces within their weights that do correspond with concepts. Probably you don't understand what concepts are -- that abstractions and concepts are basically forms of compression that let you treat different things as the same thing -- so a different way to arrive at knowing that this would be true is to consider a dataset with less parameters than there are items in the dataset and notice that the model must successfully compress the dataset in order to complete its objective.

thread

crystal_revengeNov 2024

Statistical Rethinking is an immensely practical book, and probably the best book for anyone interested in the practice of statistics.

However, it is a bit too cautious about scaring readers away from the details of how things work. Honestly, I disagree with the parent that it's a bad book for the more mathematically inclined, since I can't think of any other book that gets you solving practical problems faster. But, if you have a strong math (or computational) background, you will be craving a deeper look under the hood.

ET Jaynes' Probability Theory: the Logic of Science is, imho, the best book for someone who wants to really understand the theory and reasoning behind statistics and is comfortable lots of mathematical thinking.

For a more practical (than Jaynes) but still more detailed book on statistics then I would recommend Bayesian Modeling and Computation in Python. Not quite as easy reading as Statistical Rethinking but there will be no mystery as to what's happening.

thread