The LTTng project's blog covers its open-source tracing framework for Linux, with posts on latency analysis, releases, and Userspace RCU.
HN commenters recommend LTTng for correlated kernel and userspace tracing, and one says they are very impressed by it. It is often listed among tracing tools.
I think [this documentation page][1] does a much better job than the project homepage describing why you’d use lltng over other, related tools:
> The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, a file format optimized for the production and analyses of multi-gigabyte data.
So it has kernel hooks as well as user space libraries for a number of different programming language runtimes like Java and Python.
Compare to eBPF, which by the way is also open-source, since it’s a kernel feature. From lltng’s documentation:
> eBPF: A subsystem in the Linux kernel in which a virtual machine can execute programs passed from the user space to the kernel.
>
> You can attach such programs to tracepoints and kprobes thanks to a system call, and they can output data to the user space when executed thanks to different mechanisms (pipe, VM register values, and eBPF maps, to name a few).
[1]: https://lttng.org/docs/v2.13/#doc-lttng-alternatives
That's a pretty great overview. I've learned SystemTap myself because that's what's easy to get going in my environment. I was very surprised once I "got" it. It can do pretty much anything you'd ever need. I haven't needed to look at anything else yet, although I've been meaning to play with the LTTNG tools, as I like the idea of setting up the experiment just once, recording everything, then investigating afterwards. This [1] is a good walkthrough of that workflow.
[1] http://lttng.org/blog/2015/02/04/web-request-latency-root-cause/
This article is pretty thin but it's not wrong.
If you're interested in consistent low latency you do need to avoid TLB misses, and also page faults, cache contention, cache coherency delay (making sure no other cores are accessing your memory) from the CC protocol (MOESI/MESI(F)) and mis-prediction, and that's after you have put all your core's threads into SCHED_FIFO. Using https://lttng.org/ can be really helpful in checking what's happening.
Interesting, would it make sense to store/convert the trace in the CTF format[0]? Would the viewers from LTTng[1] be useful in analyzing this trace data?
[0] https://www.efficios.com/babeltrace
[1] http://lttng.org/files/lttv-doc/user_guide/c42.html#mainwindow
An option for correlated kernel and userspace tracing on Linux is LTTng. [0] For a comparison of LTTng and other tracers, check out the LTTng docs. [1]
[0] http://lttng.org/
[1] http://lttng.org/docs/v2.10/#doc-lttng-alternatives
To provide some context for this release announcement, the LTTng documentation has a nice intro to tracing and an explanation of how LTTng relates to other Linux tracers.
https://lttng.org/docs/v2.13/#doc-nuts-and-bolts