Elements of Information Theory

What Hacker News says

1. jeyApr 2026
   Try the textbook Elements of Information Theory by Cover and Thomas (2006)
   thread
2. SideQuarkJan 2025
   You seem wholly confused about the concept of information. Have you had a course on information theory? If not, you should not argue against those who’ve learned it much better. Cover’s book “Elements of information theory” is a common text that would clear up all your confusion.

   The “information” in a sequence of symbols is a measure of the “surprise” on obtaining the next symbol, and this is given a very precise mathematical definition, satisfying a few important properties. The resulting formula for many cases looks like the formula derived for entropy in statistical mechanics, so is often called symbol entropy (and leads down a lot of deep connections between information and reality, the whole “It from Bit” stuff…).

   For a sequence to have infinite information, it must provide nonzero “surprise” for infinitely many symbols. Pi does not do this, since it has a finite specification. After the specification is given, there is zero more surprise. For a sequence to have infinite information, it cannot have a finite specification. End of story.

   The specification has the information, since during the specification one could change symbols (getting a different generated sequence). But once the specification is finished, that is it. No more information exists.

   Information content also does not care about computational efficiency, otherwise the information in a sequence would vary as technology changes, which would be a poor definition. You keep confusing these different topics.

   Now, if you’ve never studied this topic properly, stop arguing things you don’t understand with those who’ve learned do. It’s foolish. If you’ve studied information theory in depth, then you’d not keep doubling down on this claim. We’ve given you enough places to learn the relevant topics.

   Show morethread
3. esafakDec 2023
   I think the author should study probability- and information theory if he's serious about compression. It is mathematically deep, and goes into interesting tangents like information geometry and quantum information theory.

   https://www.wiley.com/en-us/Elements+of+Information+Theory%2...

   thread
4. noudJun 2022
   Professional mathematician (but in business, not academia). I hold a PhD in mathematical physics. Below the list of books that taught me the most. However, these books are often not directly related to my current work. Also, I don't think they will be useful for everyone, as some of them are strong specialized books. These books mostly taught me to think like a mathematician. One of the strongest skills I learned as a mathematician is to dive deep into a topic and learn almost everything that you can learn about it (going from 80% of knowledge to 99.9%). I "read" these books completely several times in my career.

   * Atiyah, Macdonald - Introduction to Commutative Algebra.

   * Bourbaki (in French).

   * Gasper, Rahman - Basic Hypergeometric Functions.

   * Hasti et al - Elements Of Statistical Learning.

   * Rudin - Real & Complex Analysis.

   * Thomas, Thomas - Elements of Information Theory.

   Show morethread
5. chermiDec 2021
   You didn't ask me, but piggybacking anyway. Some of these I've been going back to for years starting back in my PhD, working through some ideas as I have time. Some I have recently been learning new things from.

   Approximately in that order below:

   Too many stat mech books, but honestly favorite is this old one. If you want a real challenge try to read to the OG Gibbs. Thermodynamics and an Introduction to Thermostatistics

   Lectures On Phase Transitions And The Renormalization Group

   Probability Theory: The Logic of Science

   Information Theory, Inference and Learning Algorithms

   Information, Physics, and Computation

   Elements of Information Theory

   Pattern Recognition and Machine Learning

   The Nature of Computation

   Principles of Cyber-Physical Systems

   Introduction to Embedded Systems, Second Edition: A Cyber-Physical Systems Approach

   Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control

   An Invitation to Applied Category Theory: Seven Sketches in Compositionality

   BTW, feel free to ask for recommendations in physics, especially stat mech, simulation/computation, condensed matter theory

   Show morethread
6. jtsukenJun 2021
   Here is a list of books for those wishing to dig deeper:

   1) The Information by Gleick is a book covering everything from biographies of the key historical figures and contributors to contemporary applications of information theory in quantum mechanics.

   2) Elements of Information Theory by Joy Thomas and Thomas Cover is a thorough and engaging textbook.

   3) Quantum information and quantum computation by Nielsen and Chuang explaining both classical and quantum information with some practical examples.

   4) Entropy and Information by Volkenstein a Soviet popular science book with math-heavy examples from biology, chemistry and physics

   and as an aside I can also recommend "Willful Ignorance: The Mismeasure of Uncertainty" by Weisberg, which is an engaging history review of probability theory and its key inventors

   Show morethread