
Casella and Berger's new edition builds the theoretical statistics from the first principals of probability theory. Thoroughly and completely, the authors start with the basics of probability and then move on to develop the theory of statistical inference using techniques, definitions, and statistical concepts.
* Statistical Inference by Casella and Berger. This book has a very good reputation for building statistics from first principles. I won't link to them, but you can find full PDF scans online with a simple search. Amazon reviews: https://www.amazon.com/Statistical-Inference-Roger-Berger/dp...
* Statistics by Freedman, Pisani, and Purves has similarly very good reviews and can be easily found online. Amazon reviews: https://www.amazon.com/Statistics-Fourth-David-Freedman-eboo...
* The majority of the Berkeley data science core curriculum books are online. This is not purely statistics but 1) is taught in a modern style that makes use of computation and randomization and 2) uses tools that may be useful to learn about.
1. https://inferentialthinking.com/chapters/intro.html (Data 8)
2. https://learningds.org/intro.html (Data 100)
3. http://prob140.org/textbook/content/README.html (Data 140)
4. https://data102.org/fa23/resources/#textbooks-from-previous-... (Data 102; this gets into machine learning and pure statistics)
The Berkeley curriculum is not the only one; there are tens, possibly hundreds, of online courses. The Berkeley curriculum is just 1) quite extensive and 2) the one I happened to read the most about when I was recently researching how data science is currently taught.
You could also look at Introduction to Probability by Joseph K. Blitzstein and Jessica Hwang (available for free here: http://probabilitybook.net (redirects to drive)).
If you want stats I highly recommend Statistical Inference by Casella & Berger -- it's extremely dry but so many stats books out there try to "make it easy" but the simplification means that you can't actually grok what's really going on. HOWEVER if you want to actually apply anything in this book you'll need to grab something more practical as well. Going through an applied stats book after having done SI is like having superpowers.
In the back of Statistical Inference by Casella and Berger, there is a great chart making similar connections between the most common statistical distributions.
There is a lot of rhyming in statistics.