Henry Schreiner

... So if the installer isn't going to ignore the version requirements, and thereby install an unsupported package that causes a breakage, then there isn't a problem with "scripts being broken because people didn't pin their dependencies". The packages listed in the PEP 723 metadata get installed by an installer, which resolves the listed (unpinned) dependencies to concrete ones (including transitive dependencies), following rules specified by the packages.

I thought we were talking about situations in which following those rules still leads to a runtime fault. Which is certainly possible, but in my experience a highly overstated risk. Packages that say they will work with `foolib >= 3` will very often continue to work with foolib 4.0, and the risk that they don't is commonly-in-the-Python-world considered worth it to avoid other problems caused by specifying `foolib >=3, <4` (as described in e.g. https://iscinumpy.dev/post/bound-version-constraints/ ).

The real problem is that there isn't a good way (from the perspective of the intermediate dependency's maintainer) to update the metadata after you find out that a new version of a (further-on) dependency is incompatible. You can really only upload a new patch version (or one with a post-release segment in the version number) and hope that people haven't pinned their dependencies so strictly as to exclude the fix. (Although they shouldn't be doing that unless they also pin transitive dependencies!)

That said, the end user…

Yes and no.

There are different types of dependencies, and there are different rules for them, but here's an overview of the best practices:

1. For applications, scripts and services (i.e. "executable code"), during development, pin your direct dependencies; ideally to the current major or minor version, depending how much you trust their their authors to follow SemVer. Also make sure you regularly update and retest the versions to make sure you don't miss any critical updates.

You should not explicitly pin your transitive dependencies, i.e. the dependencies of your direct dependencies -- at least unless you know specifically that certain versions will break your app (and even then it is better to provide a range than a single version).

2. For production builds of the above, lock each of your dependencies (including the transitive ones) to specific version and source. It is not really viable to do it by hand, but most packaging tools -- pip, Poetry, PDM, uv... -- will happily do that automatically for you. Unfortunately, Python still doesn't have a standard lock format, so most tools provide their own lock file; the closest thing to a standard we have at the moment is pip's requirements file [0].

Besides pinned versions, a lock file will also include the source where the packages are to be retrieved from (pip's requirements file may omit it, but it's then implicitly assumed to be PyPI); it can (and should, really) also provide hashes for the given packages,…

Yes, but it's (probably) the least worse thing they can do given how the "PyPI" ecosystem behaves. As PyPI does not allow replacement of artefacts (sdists, wheels, and older formats), and because there is no way to update/correct metadata for the artefacts, unless the uploader knew at upload time of incompatibilities between their package and and the upper-bounded reference (whether that is the Python interpreter or a Python package), the upper bound does not reflect a known incompatibility. In addition, certain tools (e.g. poetry) added the upper bounds automatically, increasing the amount of spurious bounds. https://iscinumpy.dev/post/bound-version-constraints/ provides more details.

The general lesson from this is when you do not allow changes/replacement of invalid data (which is a legitimate thing to do), then you get stuck with handling the bad data in every system which uses it (and then you need to worry about different components handling the badness in different ways, see e.g. browsers).

To discourage bash outside shell one-liners (where it is appropriate), fabric/invoke could be used. Bash/ssh commands are glued together using Python:

https://docs.fabfile.org/en/latest/getting-started.html#addendum-the-fab-command-line-tool

If you are Python dev, you could specify Python dependencies for script using pep 723 https://iscinumpy.dev/post/pep723/ (nix might be an overkill).

For notebooks, workflows, personal playbooks, notes Org Babel could be used (emacs) Here's example code blocks in Haskell (but many other languages can be used such as shell, jupyter, plantuml) https://youtu.be/1qOFYluebBg?si=muGfsaC1kI7Cgpyw

One of my favorite features are subtree specific settings that enable remote shell commands by configuring :dir to /ssh:host:

I wholeheartedly recommend PDM as a modern python package management tool: https://pdm.fming.dev/latest/

* greater PEP compliance with respect to poetry (and no upper
boundary constraints on packages)

* support for scripting

* parallel installation

Here is the opinion of somebody way more qualified than me:

- https://iscinumpy.dev/post/poetry-versions/

- https://iscinumpy.dev/post/bound-version-constraints/