
BPF and related observability tools give software professionals and students alike unprecedented visibility into software, helping them analyze operating system and application performance, troubleshoot code, and strengthen security. BPF Performance Tools: Linux System and Application Observability is the industry's most comprehensive guide to using these tools for observability. Brendan Gregg, author of the industry's definitive guide to system performance, introduces powerful new methods and tools for doing analysis that leads to more robust, reliable, and safer code. This authoritative guide: Explores a wide spectrum of software and hardware targets Thoroughly covers open source BPF tools from the Linux Foundation iovisor project's bcc and bpftrace repositories Summarizes performance engineering and kernel internals you need to understand Provides and discusses 150+ bpftrace tools, including 80 written specifically for this book: tools you can run as-is, without programming - or customize and develop further, using diverse interfaces and the bpftrace front-end Students will learn how to use BPF (eBPF) tracing tools to analyze CPUs, memory, disks, file systems, networking, languages, applications, containers, hypervisors, security, and the Linux kernel. Students will move from basic to advanced tools and techniques, producing new metrics, stack traces, custom latency histograms, and more.
I'm sharing a little open-source utility I wrote recently.
I'm a huge fan of Brendan Gregg's "BPF Performance Tools" book. However, every time I SSH into a fresh server, most of the diagnostic tools aren't installed there and installing them can be really annoying.
I decided to use Nix package manager and LLMs to make this process straightforward. My utility first downloads a "toolbox" of Linux utilities (built with Nix), runs Brendan Gregg's famous "60-second Linux analysis" playbook and then summarizes the results with an LLM. So "60-second Linux analysis" now becomes a single one-line command and actually takes less than 60 seconds! The utility can execute all commands in parallel and the LLM can analyze them faster than a human would.
I have a few ideas for the future, for example implementing more powerful playbooks - thanks to Nix I can easily bundle all tools I need and LLMs have no trouble analyzing outputs of tens of commands.
I'd love to get your feedback and hear any ideas you have. Thanks for checking it out.
You can launch the utility with this command:
$ curl -fsSL https://gradient.engineer/60-second-linux.sh | sh
I have pulled out a few miracles using these tools (identifying kernel bottlenecks or profiling programs using ebpf) and it has been well worth the investment to read through the books.
From there, what’s helped me most is a cycle of reading new material, building prototypes and exploring how an open source system solves similar problems. I've definitely hit that wall as systems programming can get confusing fast.
I’ve also noticed that I sometimes get stuck trying to make something perfect before I’ve even started experimenting. Forcing myself to build the lowest-effort version of an idea has been surprisingly productive. Debugging things that don’t work is frustrating, but that failure often reveals insights I wouldn’t have discovered if I were overanalyzing.
You’ve probably seen some of these resources already, but just sharing in case any of it’s useful. I work with eBPF full-time and had many similar challenges along the way, but recommend jumping back in when you have the time.
- https://www.brendangregg.com/bpf-performance-tools-book.html
And as a related activity, you could just install the bcc-tools package (on RHEL clones) and check out the /usr/share/bcc/tools directory to see what's already implemented (on latest Ubuntu, these tools seem to be installed in /usr/sbin, etc, but you could "find /usr -name *bpfcc" to get a list of eBPF tools already installed there (and test/man some more interesting ones).
For the bigger picture and other eBPF uses like networking, I'd get Liz Rice's eBPF book (free download):
- https://isovalent.com/books/learning-ebpf/
But the most valuable resource for me when I took the leap from writing bpftrace one-liners to more sophisticated modern eBPF programs was (and still is) Andrii Nakryiko's blog with examples of modern BPF programming:
https://www.brendangregg.com/systems-performance-2nd-edition...
and
BPF Performance Tools
https://www.brendangregg.com/bpf-performance-tools-book.html