Brendan Gregg

Provisioning is something that may be out of the picture you described, Terraform, Ansible, templates, whatever, more than creating that server try to recreate it after destroying it (and backups while at that). Monitoring using your own tools instead of (or along with) your provider tools will give you a view of not just when it is working wrong, but also when is working right. Security is another thing that goes both for the cloud and your own server, but for long running instances firewalls and vulnerabilities in what is exposed gets more urgent.

Regarding performance, if you want to dig in that topic, don't wait for a crisis for using those tools, try to understand how the system run, even if not under high load. Gregg have more tools (https://www.brendangregg.com/linuxperf.html) and a few great books.

I love atop - and install it everywhere that I can. The in depth metrics + historical view is excellent.

That being said, I wouldn't call it a one stop tool for performance analysis. Particularly due to the word 'analysis' - it tells you what resource was used, and what was using it. It doesn't tell you anything about /why/ the process was using it. It doesn't help with hunting down regressions. It doesn't give you insight in to what these processes are actually doing.

For that, you need to utilize strace/ltrace/ftrace/dtrace/ktap/systemtap/perf/sysdig/whatever

(If interested in the actual analysis/engineering side of performance, I recommend following Bredan Gregg's blog and buying his systems performance book - http://brendangregg.com/ )

There are two excellent books by Brendan Gregg which can teach you about eBPF:

1: Systems Performance: Enterprise and the Cloud, 2nd Edition <https://brendangregg.com/systems-performance-2nd-edition-book.html>

2: BPF Performance Tools <https://brendangregg.com/bpf-performance-tools-book.html>

Contrary to the title, I would actually recommend the first one more and only if you feel starved for more to reach for the second one. You can learn a lot about performance tools and methodologies on Linux from the first one, not just eBPF.

I've never had to recompile a Linux distro to start using eBPF btw. Most distros already use a kernel compiled with options enabling eBPF. That may not be true on some "embedded Linux" devices though.

The performance hit (~2%) is observable at runtime, assuming you're doing things which are CPU bound. For cpython programs the especially large hit (~10%) is well understood though not fixed: https://pagure.io/fesco/issue/2817#comment-826636

The flip side to this is that when you have good profiling it changes how you write and debug software. You can actually look at the whole system (eg with Flamegraphs https://www.brendangregg.com/flamegraphs.html) and find out what's taking all the time, and hopefully direct your fixes to the right place. This potentially gives you great performance benefits. Apparently Meta recompile all software with frame pointers and use perf to do this kind of analysis.

For me, BPF is the most exciting development in the kernel world for ages.

Brendan Gregg, the author of the posted blog entry, works with BPF on observability at Netflix and delivered a keynote at UbuntuMasters 2019. The video is on his blog is a great intro. [0]

I've been watching BPF for a couple of years now, and it seems slow on the uptake, but I hope Gregg is right that people will eventually start writing new drivers, firewalls, observability and security tools, loaded from userspace, but all running safely in a kernel vm, maybe even written in new async-first programming languages.

[0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-of-software.html

Great overview of the Linux network queues as provided in the Figure, should paste it on the wall somewhere.

Brendan's System Performance books provide nice coverage on Linux network performance and more [1]. It's already in the second edition, both are excellent books but the 2nd edition focuses mainly on Linux whereas the 1st edition also include Solaris.

There's also a more recent book on BPF Performance Tools by him [2].

[1] Systems Performance: Enterprise and the Cloud, 2nd Edition (2020)

https://www.brendangregg.com/systems-performance-2nd-edition-book.html

[2] BPF Performance Tools:

https://www.brendangregg.com/bpf-performance-tools-book.html