Marek Majkowski, a systems engineer, writes about Linux networking, event APIs like epoll, sandboxing, LLVM, and low-level programming.
HN commenters link his critique of epoll when comparing it with kqueue, and cite his posts on IP spoofing and HAMT memory efficiency.
To play the devil's advocate:
> In other words, back in the 90's "how do you reverse a linked list" isn't about "algorithmic thinking" or "data structures", it's _have you coded in C_. If you have, the question is trivial. If you haven't, the question was (ideally) impossible.
It is still an interesting excercise to see if someone can code reversing a string (array) and a linked-list, because they form the basic building block of many other data structures, like hash-maps and trees. I doubt it has anything to do with clang, per second, than a fizzbuzz to relatively difficult problems of solving for Graphs, for instance. So absolutely, reversing a linked-list does have everything to do with data-structures.
> Take "determine if a linked list has a cycle in it." The solution people are supposed to come up with, "Tortoise and Hare", was AFAICT published in a heavily-cited 1967 research paper. You're asking a candidate to reinvent CS research in 30 minutes!
Well, multi-threading was a decade long research, if not more. Do you then expect folks interviewing for Android Development, say, not know about critical sections, race conditions, mutual exclusion, re-entrant locks et al because these were ironed out over decades and decades of research? Guess not.
> Removed from the historical context, linked lists got rebranded as "problem solving skills". But this seems the opposite of what you'd expect from the original questions: if you're testing language familiarity, you DON'T…
Five points in reply:
1. Signalfd is a mountain of technical debt. It's not like a file at all. It reads entirely different things depending on which process is reading from a single common open file description, and it interacts with epoll in a most bizarre way, in that the process that added the signalfd to the poll set is the one whose signals will notify readiness - even if that process then closed the epoll and signalfd.
2. Signalfd is a mountain of technical debt because they built it over signals. Signals are of two forms: the one is asynchronous, the other synchronous. The one should be replaced with a generic message queue in a technical debt free design. The other is debatable. (Mach and also Fuchsia have an "Exception Port" instead, a message port someone can read from to deal with exceptions incurred by some thread.)
3. Regarding: > epoll can automatically monitor any kind of kernel resource without having to be modified, due to its composable design
Well, so can kqueue - EVFILT_READ/EVFILT_WRITE will work just as well on any file, be it a socket, an eventfd, or even another kqueue (though I don't think anyone who isn't dealing with technical debt should be adding kqueues to kqueues, nor epolls to epolls.) No need to modify kqueue! But in either case, though, you've got to modify something. Whether that's trying to fit signals into a file and getting signalfd, or whether it's adding a signal event filter and getting EVFILT_SIGNAL, something has to be…
> behind [..] tool versions
I run both Arch Linux and FreeBSD. It is not uncommon that FreeBSD's packages update faster than Arch Linux. Make sure you update your pkg config to use the latest packages instead of the quarterly packages it uses by default.
Also, I'd like to point out that FreeBSD has thousands of more packages than Fedora: https://repology.org/repositories/statistics/newest
> why would I consider FreeBSD
ZFS: Since FreeBSD isn't encumbered by the GPL, it has fully integrated ZFS which is a far better experience than on Linux. On FreeBSD ZFS always works, whereas on Linux ZFS is an out-of-tree module and your dkms builds breaks frequently when the kernel updates. Also, a lot of software on FreeBSD takes advantage of the features of ZFS, whereas on Linux most software doesn't use the features of either btrfs or ZFS.
PF: This firewall has a great config format. After using PF, Linux's iptables makes me want to walk into the ocean.
poudriere: A great and easy-to-use package builder. I use this to build both my OS and all my software with CPU optimizations for my specific processor (but FreeBSD has official package repositories, so you don't need to build your own if you don't want to.)
kqueue: FreeBSD uses the kqeueue syscall. Later Linux created their own syscall: epoll. But epoll is fundamentally broken: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/
jails: more secure than Linux containers despite being invented 7 years…
> You wanna know what else didn't ship with async? The thing all your bullshit async websocket servers run on: Linux. Last time I checked, Linux, and the POSIX world in whole, seems to be doing just fine running the entire internet without it. signal, timerfd, epoll, kqueue exist. Guess what? That's all tokio and these runtimes are doing. They can't magically put something to sleep. The kernel is the one who's not bringing your CPU to a halt when your read isn't doing anything, not tokio.
Linux since 5.11/5.15 timeframe has shipped with an growingly speedy & capable async, io-uring. There are still security issues, to the degree that Google has started simply disabling io-uring. But for many application-servers, it feels like the risk today is minimal and it feels probable the situation will improve over time; I for one would not bet against io-uring.
If anything I think there's a huge possibility to write really good languages or standard libraries that dive headfirst into this better new world. I view there as being a huge late mover advantage, ready for the taking, now that operating systems have finally gotten good, now that we can offload work so effectively to the kernel. There's works like https://github.com/tokio-rs/io-uring to make use of io-uring, but I worry that it only covers slices of what could be possible, that it's just cutting out a single path of use, and that path is deeply constrained by path dependence. It'd be interesting to see what a (not…
I feel this should really mention the epoll pitfalls (cf. https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ ) - though "fundamentally broken" is excessive hyperbole.
Also this specific article is kinda hazy on pointing out epoll's improvements over previous APIs. One half is that event subscriptions are registered/submitted for a period longer than one syscall (which the article kinda-sorta says.) But the other half is that you also have an opaque field (mostly used as pointer) that the kernel returns back to you on event notifications, which removes an extra level of lookup in userspace.
And lastly, FreeBSD has in the meantime acquired epoll support, and Linux is moving to the next-yet-again replacement in io_uring (which calling an epoll/poll replacement is an insult, it does much more.)
It's possible that wasn't the default setting on Macs back then. I don't know that cgo would be a good choice either, if you're resolving a ton of domains at once. Early versions of Go would create new threads if a goroutine made a cgo call, and an existing thread was not available. I remember this required us to throttle concurrent dial calls, otherwise we'd end up with thousands of threads, and eventually bring the crawler to a halt.
To make DNS resolution really scale, we ended up moving all the DNS caching and resolution directly into Go. Not sure that's how you'd do it today, I'm sure Go has changed a lot. Building your own DNS resolver is actually not so hard with Go, the following were really useful:
https://idea.popcount.org/2013-11-28-how-to-resolve-a-million-domains/
https://github.com/miekg/dns