Shuveb Hussain writes Linux performance tutorials, including io_uring guides, example web servers, and deep dives on Firecracker and PSI.
Commenters point to his io_uring tutorial as fantastic learning material, with examples such as the SQ polling demo using BPF.
Depends on your usage patterns. On 8GB I gave it (ZRAM) between 512MB to 768MB RAM, and later on just using whatever values for the relevant sysctls AntiX was using(or their scripts created dynamically during setup/startup) at the time. It isn't just vm.swappiness, there are others, and they changed, or got removed, and replaced by others, over the years. And all of them sort of interwoven, influencing each other via what's happening in vm-subsystem. Can have bad feedback loops if you do it wrong, and be counterproductive. So you have to experiment, and measure with different settings, for what you're running.
Where measure can be anything from htop/atop over https://github.com/cdown/psi-notify , to https://github.com/noiseonwires/memory-pressure and so on.
To understand what these sysctls are about, how they are related to each other, and to ZRAM, I'd recommend reading, or at least skimming https://chrisdown.name/2018/01/02/in-defence-of-swap.html , https://chrisdown.name/2019/07/18/linux-memory-management-at-scale.html , https://linuxblog.io/linux-performance-almost-always-add-swap-space , https://linuxblog.io/linux-performance-almost-always-add-swap-part2-zram , https://linuxblog.io/linux-performance-no-swap-space , https://linuxblog.io/running-out-of-ram-linux-add-zram , https://github.com/ValveSoftware/SteamOS/issues/899 , https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio , https://github.com/CachyOS/CachyOS-Settings/pull/19 ,…
> Interesting CGI content linked on there.
>
>I've been reading about / hacking on CGI recently, and it's been kinda fun!
>
>Question: One thing I keep reading is how inefficient it is to start a new process for each incoming connection. Could someone explain to me why that's such a bottleneck? I imagine it being an issue back when CGI was used everywhere, people moving away from CGI, and forgetting about it. But hasn't there been improvements in the meantime? Computers from today can run circles around those from a few decades back. Has everything improved except the speed / efficiency of starting a new process?
>
It's not as bad as you think it is; just change the webserver to pre-fork. From this link[1], and the nice summary table in this link[2] - I note the following:
1. pre-forked servers perform very consistently (the variation before being overwhelmed) and appears at a glance to only be less consistent than epoll.
2. For up to 2000 concurrent requests, the pre-forked server performed either within a negligible margin against the best performer, or was the best performer itself.
3. The threaded solution had the best graceful degradation; if a script was monitoring the ratio of successfull responses, it would know well beforehand that an imminent failure was coming.
4. The epoll solution is objectively the best, providing both graceful degradation as well as managing to keep up with 15k concurrent requests without complete failure.
With all of the above…
> That's not all, though, right? Unless you're using the dubious kernel-driven polling thread, which I gather is uncommon, you _also_ have to make a submit system call of some kind after you've finished appending stuff to your submit queue. Otherwise, how would the kernel know you'd done it?
Correct, the "check for new entries" system call is called io_uring_enter(). (0)
> Some system calls are not especially asynchronous: they perform some amount of work that isn't asynchronously submitted to some hardware in quite the same way as, say, a disk write may be. What thread is actually performing that work? Does it occur immediately during the submit system call, and is completed by the time that call returns? Is some other thread co-opted to perform the work?
A kernel thread. The submit system call can be optionally made to wait for completion, but by default it will always immediately return.
> Whose scheduling quantum is consumed by this?
That's a good question. The IO scheduler correctly sees them as belonging to the submitting thread, but if you issue a bunch of computation-heavy syscalls, I would not be surprised if they were not correctly accounted for.
(0) https://unixism.net/loti/ref-iouring/io_uring_enter.html#c.io_uring_enter
I've been super curious to see what was at stake here! This sounds better than I'd dared to hope for.
I kind of thought this was just gonna be some kind of deferred texture loading thing, help with streaming assets.
If it actually allows inter-warp sequencing, it sounds like it might possibly solve the chief complains supreme GUI master Raph Levien recently had in I want a good parallel computer, which so that even though we can dynamically add shaders & construct a dynamic workgraph (largely thanks to VK_AMDX_shader_enqueue?), there isn't any sequencing/fencing/barrier-ing between the sections. https://raphlinus.github.io/gpu/2025/03/21/good-parallel-computer.html https://news.ycombinator.com/item?id=43440174
Not applicable to GPUs, but since I ran into it recently, it's interesting to see how io_uring handles sequenced submissions. Here's Lord of io_uring's write-up, https://unixism.net/loti/tutorial/link_liburing.html#link-liburing
Edit: having read the article more fully, I'm not sure this is about waves depending on each other. Maybe more about them trying to access memory. Apologies. Hopefully someday!
> The security credentials of an io_uring are established at creation time.
Well, the security credentials of a file descriptor are established at ```open()``` time. Is this really so different?
Running through the list of io_uring opcodes, only two of them work on ```char*``` paths: IORING_OP_OPENAT and IORING_OP_OPENAT2, which were added to io_uring after it was introduced. All the rest operate solely upon already-opened file descriptors, and POSIX doesn't do any permission checks for operations on already-open descriptors anyways.
It sounds like the issue you mention was an oversight when IORING_OP_OPENAT(2) were added. The easy, and retroactively safe, fix is to forbid these opcodes on submission queues by default, unless the submission queue was opened with a special flag whose name includes a scary warning -- something like IOURING_I_WILL_NOT_SETUID_WITHOUT_CLOSING_THIS_RING_FIRST.
https://unixism.net/loti/ref-iouring/io_uring_enter.html
Good article. However, it would be interesting to compare these two things against better system calls.
> the data referred to by these pointers must be copied into the kernel
Modern OS kernels avoid that when they can. The article is about Linux, modern ones have io_uring: https://unixism.net/loti/what_is_io_uring.html
In short, that thing allows to setup these pointers in advance, and make the memory visible to both user-mode code and OS kernel at the same time. This way no need to copy the data.
Unfortunately, implementing that thing is way more complicated than calling read() or setting up memory mapping. For optimal performance you gonna need to zero-copy all the way up to whatever code is consuming the data read from these files.