World-renowned software performance expert Richard Sites offers expert methods and advanced tools for understanding complex time-constrained software dynamics, improving software reliability, and troubleshooting challenging performance problems. He addresses CPU, memory, disk, network, and more, helping students fix programs that are always too slow, or work well at times but inexplicably slow down at other times. Sites draws on several decades of experience pioneering software performance optimization, and extensive experience teaching developers at the graduate level. He introduces principles and techniques which can be used in any environment, from embedded devices to datacenters, illuminating them with examples based on x86 or ARM processors running Linux and linked by Ethernet. Sites also guides students through building and applying a powerful new extremely-low-overhead open source software tool, KUtrace, to trace exactly what is executing on every CPU core. Using insights gleaned from this tool, students can apply nuanced approaches to performance enhancement--not merely brute-force techniques such as turning off caches or cores. "In the Preface, Sites mentions that he got many helpful suggestions while teaching graduate-level courses after retiring from Google... I think that any senior CS student... can benefit by reading this book.
My contribution to the discussion:
Something you could use to debug issues / detect weak points is a tracer.
This book by Dick Sites (Early Google Engineer) goes over the design and use of such a tracer: Understanding Software Dynamics (review: https://www.usenix.org/publications/loginonline/understandin...)
KUtrace is absolutely one of the most powerful tools I've used for deeply understanding performance bottlenecks (after isolating issues) such as poor scheduling behavior. I would highly recommend reading his book "Understanding Software Dynamics" [1] if you are interested in learning more about KUtrace or performance bottlenecks/optimizations in general. The book is quite dense and dives deep into the performance characteristics of many examples of the five fundamental resources (according to Dick): CPU, Memory, Disk/SSD, Network, and Software critical sections.
[1]: https://www.oreilly.com/library/view/understanding-software-...
He discusses exactly what you're describing (L1/2/3 cache hit rates, their performance implications, how compiler optimizations can fool us into thinking we have a good hit rate, etc).
Also take a look into Intel VTune and Processor Tracing to understand how performance counters like Instructions per Cycle are calculated.
https://www.amazon.in/Understanding-Software-Addison-Wesley-...
This is a recent book and I have not completed reading it but I know already is it destined to become a classic. If you are serious about learning how to understand and improve code, this is the book for you. (Full disclosure: Dick is a long-standing friend and one of the brightest and insightful guys around.)
Writing Efficient Programs (Prentice-Hall Software Series) 1st Edition by Jon Louis Bentley (https://www.amazon.com/Writing-Efficient-Programs-Prentice-H...)
This book is out of print but I believe it's a better read that Jon's Programming Pearls books. C is a great (and permissive) programming language. Jon shows how the language can be exploited for gain performance. Evil but effective.
Hacker's Delight 2nd Edition by Henry Warren (https://www.amazon.com/Hackers-Delight-2nd-Henry-Warren/dp/0...)
The Hacker's Delight is exactly what Hank Warren intended--a collection of tricks and facts that programs can exploit. It's informative on many levels. It uses deep knowledge of how numbers (and other mathematical objects) work to compute useful information. And it shows how seemingly arcane information can be useful. For a programmer excited by the fabric of programs, it is a continual delight.
The first edition has a clarity and compactness that I find appealing. The second edition has additional material. When you get through all of the Delights you can read Knuth Volume 4 which embeds more arcane and useful knowledge.