Facebook's open source microsites document Linux kernel work such as Btrfs, BPF CO-RE, BTF type information, pressure metrics, and oomd.
Commenters cite it as a reference for BPF portability, BTF, and memory pressure metrics when discussing Linux kernel and cgroup behavior.
Depends on your usage patterns. On 8GB I gave it (ZRAM) between 512MB to 768MB RAM, and later on just using whatever values for the relevant sysctls AntiX was using(or their scripts created dynamically during setup/startup) at the time. It isn't just vm.swappiness, there are others, and they changed, or got removed, and replaced by others, over the years. And all of them sort of interwoven, influencing each other via what's happening in vm-subsystem. Can have bad feedback loops if you do it wrong, and be counterproductive. So you have to experiment, and measure with different settings, for what you're running.
Where measure can be anything from htop/atop over https://github.com/cdown/psi-notify , to https://github.com/noiseonwires/memory-pressure and so on.
To understand what these sysctls are about, how they are related to each other, and to ZRAM, I'd recommend reading, or at least skimming https://chrisdown.name/2018/01/02/in-defence-of-swap.html , https://chrisdown.name/2019/07/18/linux-memory-management-at-scale.html , https://linuxblog.io/linux-performance-almost-always-add-swap-space , https://linuxblog.io/linux-performance-almost-always-add-swap-part2-zram , https://linuxblog.io/linux-performance-no-swap-space , https://linuxblog.io/running-out-of-ram-linux-add-zram , https://github.com/ValveSoftware/SteamOS/issues/899 , https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio , https://github.com/CachyOS/CachyOS-Settings/pull/19 ,…
> If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on.
This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer.
Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly.
An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory.
On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages:
* allows you to…
Most examples of BPF code are written in a mix of Python and C using BCC, the "BPF Compiler Collection", which essentially treats all of LLVM and clang as a library callable from Python code.
I can't get my head around using it that way, and have found it pretty straightforward to just write C programs, compiled with clang `-target bpf`. Until very recently, writing anything interesting this way required you to declare all functions inline, compile into a single ELF .o, and, of course, avoid most loops. But most of the kinds of things you'd write in BPF tend not to be especially loopy (you can factor most algorithmic code out into userland, communicating with BPF using maps).
A big issue for this kind of development is kernel compat; struct layouts can change from release to release, for instance. This isn't a problem for us at Fly, because we just run the same kernel everywhere, but it's a real problem if you're trying to ship a tool for other people's systems. But that's changing with CO-RE; recent kernels can export a simplified symbol table in a BPF-legible format called BTF, and the leader can perform relocations. Facebook has written a bunch of good stuff about this:
https://facebookmicrosites.github.io/bpf/blog/2020/02/20/bcc-to-libbpf-howto-guide.html
We published this paper "TMO: Transparent Memory Offloading in Datacenters" last year which covers some Linux memory management mechanisms that may be quite useful for providing reasonable estimations to application memory usage.
We observed that the real memory footprint for applications depends on many factors: file access pattern, disk IO speed (especially if swap is enabled), ssd vs hdd, application latency sensitivity, etc. Instead of coming up with some overly complicated heuristic, we use the Linux kernel provided memory.pressure [0] metric via cgroup v2. It measures the amount of time spent waiting for memory (page fault etc). Then by slowly reclaiming memory from the application until its memory pressure hits some target (say 0.1%), we can claim that the steady state usage is the actual memory footprint.
This may not be useful for PC but could be very useful for data center to track memory regression, and also to harvest disk swap without concerning too much about the cliff effect when the host runs out of memory and suddenly kernel pushes everything to swap space.
[0] https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.html
I mean think of low priority service as services that are not latency sensitive (background jobs).
you want those low priory cgroup to use the extra memory if it’s available (not used by a high priority cgroup)
it a high priority cgroup need the memory it steal it from the low priority cgroup up to the minimum guaranteed to this low priority cgroup.
this low priority cgroup memory contention will turns into IO pressure from page faults. Then IO limit on the low priority cgroup will cap how much IO it can generate by throttling it.
Meanwhile the high priority cgroup use it’s guaranteed cpu, guaranteed IO and guaranteed memory with no hiccup.
to learn more check « fbtax2 memory controller configuration » at https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html
That's once more not an answer to my question…
Regarding spontaneously reboots: I never ever experienced this on Linux besides once having a hardware defect. (I would check the RAM in your case first; Manjaro is an Arch derivative so it incorporates the latest SW but it's not so terribly unstable that it would crash every two month I guess; not sure of course. I'm avoiding Arch for exactly the reason not being stable enough cause it's extremely fresh SW).
Regarding the unresponsiveness on low memory: That's an known problem caused by a live-lock in the kernel. It has a few solutions by now. Just have a look at Facebooks oomd (or one of the solutions working in a comparable way):
https://facebookmicrosites.github.io/oomd/docs/overview