Sam Boyer, a developer who worked on Go dependency management, writes about how package managers are designed and why they are hard.
HN commenters repeatedly cite "So you want to write a package manager" as the go-to explanation of the problems, theory and ecosystem issues involved.
> Most package managers have lockfiles
> Locking dependencies is, generally, a solved problem across most ecosystems
As someone who is building a package manager for work, and has looked at pretty much every package manager out there (and their ecosystem adoption), I can only say that those don't reflect the current reality of package management (no matter how much I wish it were true).
Bundler was the first mainstream package manager to adopt a lockfile (AFAIK) a mere 12 years ago. Many many language ecosystems predate that and are still lacking lockfiles (or even widespread adoption of a single compatible package manager).
NPM only got lockfiles 5 years ago (after being pressured by yarn). Gradle got them less than 3.5 years ago, and Maven still doesn't have them (though a niche plugin for it exists). The Python ecosystem is still a hot mess, with ~3 competing solutions (Poetry, Pipenv, Conda), of which Conda just got a 1.0 of their decent conda-lock incubation project a month ago, but due to how setuputils works, the cross-platform package management story is broken almost beyond recovery. In Conan lockfiles are still an experimental feature today.
I could go on and on, but I hope that I could paint a picture that while one could argue that with the advent of lockfiles, locking dependencies has become a solved problem _conceptually_, the current status of implementation across ecosystems is still horrible. I'm also constantly amazed about how little love is put into…
>, but what is there to be opinionated about with a build system?
I think you're saying this because your idea of a "build system" is a very simple set of sequential steps from known source code input files to an output binary.
The differing opinions come in when the "build system" includes various contradictory philosophies of how to configure and specify the building of complex software.
Different opinions on:
- syntax : should build config be XML, or JSON, or YAML, or custom syntax? E.g. Ant and Maven used XML but Gradle does not
- dependencies search: should build system implicitly find and add relevant dependencies? Or should programmer explicitly specify each one?
- reproducible builds (version pinning) vs auto-updated dependencies.
- should the build system be "smart" about cross-platform differences? If yes, you end up with complicated build tools like GNU Autoconf "configure" bash script, or CMake's complex syntax.
- should build system do optimization and bundling tasks that's not strictly limited to "compile" steps? Some Javascript build tools try to eliminate redundant or unused js code to make downloads smaller.
- etc, etc.
I also recommend reading the blog post "So you want to write a package manager"[1] to get an idea of how complicated a build system can get. The title says "package manager" but much of the material is also about build systems.
The bottom line is that reasonable people can disagree on the priorities and therefore, you can't…
> I'm interested in your view about this very aspect.
I'm conflicted. I personally waver between "stallman is always right" and "free software misses the point" near constantly, myself. With regards to WebAssembly, it's also complicated. You can argue that JavaScript is basically in the same place, and so for me, it's kind of a moot point. That is, there's no requirement for JavaScript to be free software today either. In practice, obfuscated/minified JS and WebAssembly are basically the same, with the exception that the tooling for de-{obfuscating,compiling} JS is currently more robust and mature since it's been around a lot longer.
TL;DR: I don't think that WebAssembly really changes the game here, for good or for bad.
> I didn't mean to restrict this to Apt
Quite fair! And as I alluded to above, it's not just the thing that I posted. This is quite long but I have little time, so I'm gonna be brief. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 basically is, IMO, the gold standard for talking about these things. However, I think the distinction of OS/system package manager vs Language package manager is important. Basically, Cargo and Apt (I'll follow your lead here with using Apt as a standin) have very different goals, and very different constants. This is because they're doing two very different jobs. Specifically, system package managers are about providing a unified set of packages to end-users. Language package managers are for…
Depends on the use. For a library, it doesn't need to be distributed at all, since the application that's using the library will eventually dictate the actual versions of dependencies (so it can play nicely with other libraries). For an application, yes: you commit it along-side the pipfile, and re-generate the lock file when you upgrade things.
https://caremad.io/posts/2013/07/setup-vs-requirement/ covers it pretty well. Or https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 for a fantastic read and a LOT more context.
Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management.
https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.pdtxz22hv
Discussion https://news.ycombinator.com/item?id=11088125
Sam is currently working on a library for Go https://github.com/sdboyer/gps
Semi related but recently Sam Boyer wrote a good article about writing a package manager. He explores the topic quite well, indicating that there are several things that are conflated as package managers
https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.740o43vxi