Hongli Lai, a developer who writes about Ruby memory use, Linux tracing with eBPF, distributed locking, packaging and server monitoring.
HN commenters cite it as an interesting, detailed resource, especially the Ruby memory bloat post, the eBPF explainer and the Google Cloud Storage locking algorithm.
I also wrote about distributed locking a while ago, particularly about implementing one on Google Cloud Storage. https://www.joyfulbikeshedding.com/blog/2021-05-19-robust-distributed-locking-algorithm-based-on-google-cloud-storage.html
This is useful because cloud storage is very cheap and serverless. It certainly beats running a Redis or PostgreSQL instance.
In my research and implementation I took care of the problems described in this article (as well as the problems I encountered in other distributed lock implementations).
Clients either freezing or outright crashing or disappearing is definitely a major problem. So timeouts are a must. But you can't just have a timeout because you don't know whether a client is arbitrarily delayed or whether it has disappeared. Furthermore, some workloads have an unknown completion time so you can't pick any sensible timeout.
I address this by encouraging clients to behave as follows:
1. Make the timeout relatively short, but regularly refresh your lease to let others know that you're still alive.
2. Your lease may at any time be taken over by another client that thinks you've disappeared. So you must regularly check whether your lease on the lock is still valid. This checking must be done way, way before your timeout expires, in order to account for the possibility that you can be arbitarily delayed at any moment (GC pauses, scheduling pauses, network delays, etc). I recommend checking in an interval at most 1/8 of the timeout.…
> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies.
Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1].
See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html for a recent basic ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
[0] http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html
[1] that's Brendan Gregg saying it: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html — having contributed to it and being one of the leading dtrace experts
One thing that has been known for a while is that there are allocation patterns where LibC malloc doesn't give back a sufficient amount of pages even when the pages are clean.
See for example https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html#a-magic-trick-trimming . In the end I think the consensus was that jemalloc was just better than invocing malloc_trim, but invoking malloc_trim now and then can certainly be a lot better than using neither malloc_trim or jemalloc.
> At this point jemalloc is so good
Some folks disagree:
> [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy.
via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-status-of-ruby-memory-trimming-and-how-you-can-help-with-testing.html
It’s definitely surprising that macOS does this, but it makes me also wonder if the authors considered just modifying the binary so that the dynamic linker could find the library: https://www.joyfulbikeshedding.com/blog/2021-01-13-alternative-to-macos-dyld-library-path.html
Not sure if this is jargon-free enough, but maybe it helps: https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html