Rich Felker

I have been using Musl for various purposes for quite some time. I think my favourite things are:

* The team are very helpful, and fix things fast and correctly. You can see some of the detail in this post on race conditions in glibc http://ewontfix.com/16/

* It is very standards compliant, so it is a good portability test for your code. It is a bit like using a BSD libc, except it does have Linux specific syscalls etc.

* Static linking works, which it doesnt properly in glibc. If you want to be like those trendy Go people and provide simple statically linked single file binaries, for running in containers etc then you can. And they will be small: the glibc shared library is 1.8MB (not including rt, pthread etc), while a Musl statically linked copy of Lua for example is only 160k. Static linking is very useful for all sorts of things, my most recent use was cross compiling bianries to run under qemu-user for testing.

* It is BSD licensed so you can use it in completely different projects however you want, eg OSv uses it http://osv.io/

* The code is small and readable. If you want to know how things work you can just take a look, quickly find the code and understand it. Just finding the code in glibc takes ages. I keep a reference copy just for this purpose.

You can use the "cross compiler" musl-gcc that ships with it to build if your code is happy being cross compiled (should be, most things just work). If you have a lot of dependencies it might be easier to use a…

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway.

* Blogs with a math focus:

http://blog.sigfpe.com

http://bit-player.org

http://mathr.co.uk/blog/

https://www.johndcook.com/blog/

* Blogs with a programming languages focus:

https://existentialtype.wordpress.com

https://bluishcoder.co.nz

http://blog.ezyang.com

http://akkartik.name

http://lambda-the-ultimate.org

http://mrale.ph

* Blogs with a UI design focus:

http://www.brandonwalkin.com/blog/

http://ignorethecode.net/blog/

http://interuserface.net

* Blogs with a graphics / 3D rendering focus:

http://c0de517e.blogspot.com

http://diaryofagraphicsprogrammer.blogspot.com

https://directtovideo.wordpress.com

http://graphicrants.blogspot.com

http://www.reedbeta.com

http://www.realtimerendering.com/blog/

https://blog.demofox.org

http://eelpi.gotdns.org/blog.wiki.html

* Other interesting / more general programming blogs (many of these have a "systems" focus):

http://danluu.com

http://ewontfix.com

http://chneukirchen.org/blog/

http://neverworkintheory.org

http://apenwarr.ca/log/

https://www.snellman.net/blog/

https://www.pvk.ca

http://psy-lob-saw.blogspot.com

https://randomascii.wordpress.com

http://fastcompression.blogspot.com

https://fgiesen.wordpress.com

http://blog.plover.com

http://chneukirchen.org/trivium/

http://rachelbythebay.com/w/

I think the proper title is: Linux Process and Threads Don't Mix.

The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!)

This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse.

Check out https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.

> In general, I have found that things which are difficult on Linux are so because the problem is difficult [...].

Hard disagree. Audio mixing is not difficult[1]. The Linux kernel guys were right - it does not belong in the kernel. The userspace story however, has been a complete shitshow for decades. I think Pipewire mostly fixed that? Not sure, sometimes I still have to log out and back in to fix audio.

The funniest part? It's been working in the BSDs all along. I recommend reading the source of sndiod[1].

[1]: <https://cvsweb.openbsd.org/src/usr.bin/sndiod/>

What's even worse? Probably systemd. I try not to hold a strong opinion - I tolerate it, the way I tolerate traffic when taking a walk. The technical issue however is several orders of magnitude simpler - again, the BSDs come to mind, but you can also write a complete PID1 program in about 20 lines of straightforward C[2]. I don't mind the commands being unfamiliar (they're already all different in almost every OS family); it's that the entire package is dreadfully large in scope, opaque, and I find it more difficult to interact with than anything else in this landscape.

[2]: <https://ewontfix.com/14/>

OTOH, Linux's setuid and similar interfaces are per-thread, not per-process as required by POSIX and as what most people would expect--particularly in the context of software security. glibc and musl have to emulate the POSIX behavior, and doing it correctly is extremely tricky. See https://ewontfix.com/17/

Linux's various APIs used for container frameworks also tend to be per-thread, not per-process; that can be both useful and a giant headache, depending on context. It's somewhat ironic that Go was used for Docker as Go is pretty much the worst possible language you could choose in this regard. Go deliberately and thoroughly obscures native thread and process semantics. IIRC, Docker was already well along and established before Go even introduced an API for pinning a goroutine to a machine (kernel) thread. I once ran across a comment where the author of (I think) runc lamented his choice of Go. But I haven't been able to find it again, so it's entirely possible it's a misattribution on my part.

> musl libc

I'm conflicted on Musl. I like the idea. I like the focus on robustness. I like Felker's style [1]. But for God's sake, Musl refuses to implement basic operating system primitives we've had for decades, like dlclose(3), all because the author, from his necessarily limited perspective, just can't understand why someone would want those facilities.

I'm sticking with glibc until the Musl people decide they want robustness and feature parity.

By the way: PAM isn't "add in the bloat of additional backends". Dynamic loading of backends is the opposite of bloat.

There's absolutely no reason Felker can't greatly improve libc robustness and maintain at least source-level compatibility with a libc we've used for decades and that has many valuable features. Musl's sin is being gratuitously opinionated.

[1] http://ewontfix.com/, good for the most part. Wrong about 32-bit PIC being terrible. (What we really need is -fvisibility=hidden and -Bsymbolic by default.)