
From there, what’s helped me most is a cycle of reading new material, building prototypes and exploring how an open source system solves similar problems. I've definitely hit that wall as systems programming can get confusing fast.
I’ve also noticed that I sometimes get stuck trying to make something perfect before I’ve even started experimenting. Forcing myself to build the lowest-effort version of an idea has been surprisingly productive. Debugging things that don’t work is frustrating, but that failure often reveals insights I wouldn’t have discovered if I were overanalyzing.
You’ve probably seen some of these resources already, but just sharing in case any of it’s useful. I work with eBPF full-time and had many similar challenges along the way, but recommend jumping back in when you have the time.
https://www.packtpub.com/authors/kaiwan-n-billimoria
I’m halfway into their first Linux Kernel Programming book and like it so far.
1. Systems Operations is first and foremost about understanding systems, in all of their complexity, which means understanding the internals of your OS primarily.
2. Performance and networking, in particular, are super important areas to focus on understanding when it comes to learning the topic to help with software development.
3. A lot of it is about understanding concepts in abstract and being able to extrapolate to other situations and apply these concepts, so there's actually quite a lot of useful information that can be learned on one OS and still applied to another OS (or on one game engine and applied to another, et al).
Here's a few books I think are worth reading, not in any particular order of prevalence, but loosely categorized
Databases:
High Performance MySQL: https://www.amazon.com/gp/product/1449314287/
SQL Queries for Mere Mortals: https://www.amazon.com/gp/product/0321992474/
The Art of SQL: https://www.amazon.com/gp/product/0596008945/
Networking:
TCP/IP Illustrated: https://www.amazon.com/exec/obidos/ISBN=0201633469/wrichards... (updates on author's site at http://www.kohala.com/start/tcpipiv1.html)
The TCP/IP Guide: https://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-P...
UNIX Network Programming: https://www.amazon.com/dp/0131411551
Beej's Guide to Network Programming: http://beej.us/guide/bgnet/
Operating Systems:
Operating Systems Concepts: https://www.amazon.com/Operating-System-Concepts-Abraham-Sil... (various editions, I have the 7th edition... I recommend you find the latest)
Modern Operating Systems: https://www.amazon.com/Modern-Operating-Systems-Andrew-Tanen... (the "Tanenbaum Book")
Operating Systems Design and Implementation: https://www.amazon.com/Operating-Systems-Design-Implementat-... (the other one, the "MINIX Book")
Windows Internals:
Part 1: https://www.amazon.com/Windows-Internals-Part-architecture-m...
Part 2: https://www.amazon.com/Windows-Internals-Part-2-7th/dp/01354... (I had the pleasure of being taught from this book by Mark Russinovich and David Solomon at a previous employer, was an amazing class and these books are incredible resources even applied outside of Windows, we used 5th edition, I linked 7th, which has the 2nd part pending publication).
MacOS Internals:
Part 1: https://www.amazon.com/MacOS-iOS-Internals-User-Mode/dp/0991...
Part 2: https://www.amazon.com/MacOS-iOS-Internals-II-Kernel/dp/0991...
Part 3: https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp...
Linux Kernel Programming:
Part 1: https://www.amazon.com/Linux-Kernel-Development-Cookbook-pro...
Part 2: https://www.amazon.com/Linux-Kernel-Programming-Part-Synchro...
The Linux Programming Interface: https://www.amazon.com/Linux-Programming-Interface-System-Ha...
General Systems Administration:
Essential Systems Administration: https://www.amazon.com/gp/product/0596003439/
UNIX and Linux Systems Administration Handbook: https://www.amazon.com/UNIX-Linux-System-Administration-Hand...
The Linux Command Line and Shell Scripting Bible: https://www.amazon.com/Linux-Command-Shell-Scripting-Bible/d...
UNIX Shell Programming: https://www.amazon.com/Unix-Shell-Programming-Stephen-Kochan...
BASH Hackers Wiki: https://wiki.bash-hackers.org/
TLDP Advanced BASH Scripting Guide: https://tldp.org/LDP/abs/html/
The Debian Administrator's Handbook: https://debian-handbook.info/browse/stable/
TLDP Linux System Administrator's Guide: https://tldp.org/LDP/sag/html/index.html
Performance & Benchmarking:
Systems Performance: https://www.amazon.com/Systems-Performance-Brendan-Gregg-dp-... (this is Brendan Gregg's book where you learn about the magic of dtrace)
BPF Performance Tools: https://www.amazon.com/Performance-Tools-Addison-Wesley-Prof... (the newer Brendan Gregg book about BPF, stellar)
The Art of Computer Systems Performance Analysis: https://www.cse.wustl.edu/~jain/books/perfbook.htm (no longer available from Amazon, but is available direct from publisher. This is basically the one book you should read about creating and structuring benchmarks or performance tests)
I guess that's a "reading list", but this is just a small part of what you need to know to excel in systems operations.
I would say for the typical software developer writing web applications, the most important thing to know is how databases work and how networking works, since these are going to be the primary items affecting your application performance. But there's obviously topics not included in this list that are also worth understanding, such as browser/DOM internals, how caching and CDNs work, and web-specific optimizations that can be achievable with HTTP/2 or QUIC.
For the average software developer writing desktop applications, I'd say make sure you /really/ understand OS internals... at the base everything you do on a computer system is based on what the OS provides to you. Even though you are abstracted (possibly many layers) away from this, being able to peel back the layers and understand what's /really/ happening is essential to writing high-quality application code that is performant and secure, as well as making you a champ at debugging issues.
If you're trying to get into systems operations as a field, this is just a brush over the top surface and there's a lot deeper diving required.
While it's still based on the 2.6 kernel, it's probably good enough for the basics. Not sure there is a more modern source.
(Robert Love's Linux Kernel Programming was also a nice introduction to how the Kernel works, though it might be outdated in some areas now as well)
* Design and implementation of the FreeBSD operating system: Good and thorough deep dive in FreeBSD OS. Must have.
* FreeBSD Device Drivers: A Guide for the Intrepid: Didn't quite read it all but looks fine for FreeBSD.
* Mac OSX internals - a systems approach by Amit Singh: It was good back in days but now outdated.
* Linux Device Drivers, 3ed : Very dated but still good to grasp Linux device drivers in general. The code examples are bit silly but good enough. There are some GitHub repos which have updated code for latest kernels.
* Essential Linux Device Drivers by S Venkateswaran: It complements LDD3 book quite nicely. Has some real device examples and very exhaustive. Must have.
* Linux kernel development by Robert Love: Very good for short/quick intro. Best for preparing before interviews ;)
* Professional Linux kernel architecture by Wolfgang Mauerer: As other book, its dated but some of the explanations about interrupts, PCI, etc are good. His callgraph approach was very handy in understanding things.
* The Linux programming interface by Michael Kerrisk: Not really about OS but next thing close to it - system programming (the real system programming). Must have.
* Understanding Linux virtual memory manager by Mel Gorman: As other books, it's dated but still one of the best available to get a handle on memory management under Linux. Must have.
* Understanding the Linux kernel: Dated but still my go to book to refresh certain subsystems. Must have.
* Linux Kernel Programming by Michael Beck: Mentioned for historical reasons, else most out dated book here (2.4 based). Horrible editing and English but heck! I loved it back in the days :)
* Linux kernel networking by Rami Rosen: Never read it but quite dated.
* Understanding Linux network internals by Christian Benvenuti: A real bible of Linux networking. If nothing else, your jaw drops at the effort author made to write this book. Dated but unlike more generic Linux kernel things, network stack is still same in its core. Must have.
* The Linux kernel primer: A top-down approach for x86 and PowerPC Architecture: Very dated book but good to read from PPC perspective. Lot of things have changed since its publication.
* See MIPS Run by Dominic Sweetman: dated but gives good idea about MIPS internals and how Linux uses it.
* IA-64 Linux Kernel: Design and Implementation: Its dated not just w.r.t code but also w.r.t. hardware. Nonetheless, it gives a good insight into IA-64 architecture and Linux from non-x86 perspective.
* Definitive guide to Xen hypervisor: this is the only book on virtual machine which is not just from user perspective. While best way to learn VM is through reading architecture spec and code, this book still satisfied me w.r.t. virtual machine internals.
Every other book on amazon about Linux kernel is more or less useless. For more academic book, "Operating systems - three easy pieces" is good.