Donald Stufft, a Python packaging maintainer, writes about PyPI, package signing, and dependency management for Python.
Commenters split on his package signing post: some call it insightful on why signatures don't fix security, while one says it proposes nothing.
The vast majority of python codebases are still in python 2, even though many of the popular framework have been ported to 3, and there's no reason for most of them to bother with the amount of work it can be to port to py3 when they're happy with py2. Whether it be devs like Dropbox or open source software like NVDA, they will stay in the world of python 2 for the years to come. The amount of customers they could get by supporting python 3 is definitely not worth the effort unless they get paid explicitly for it.
In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real.
See also:
https://caremad.io/2015/04/a-year-of-pypi-downloads/
https://www.reddit.com/r/Python/comments/45sm94/what_are_the_most_recent_python_3_vs_python_2/czzy5wa
"Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%"
I expect things…
As I mentioned the last time this came up, I recently converted a medium-sized (about 30,000 lines) system from Python 2 to Python 3. The problem is not the language changes. Those are minor, and most of the new forms worked in Python 2.7.9. It's the state of the external modules. Important old Python 2 modules have been discontinued, and their replacements are not used enough or tested well enough to use reliably. I kept finding bugs that should have been found in the first months of heavy use of a module.[1] It took an extra month of work to find and work around all the external module bugs. These aren't obscure modules; they're ones most large server-side applications would need.
Porting was at least possible. Two years ago, I took a look at doing this, and there weren't enough modules ready to even attempt it. Now there are. There is progress. Slow progress.
The download statistics tell the story: "We can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%."[2] That's why the external modules don't work reliably on Python 3. 95% of the package use is on Python 2. Python 2 use is probably higher than that, since those are download statistics; PyPI doesn't see packages downloaded long ago and still running.
There's extensive denial about this among Python fans. That's a big part of the problem. Another problem is that PyPI is just a link farm - unlike CPAN, it doesn't centralize code hosting, bug…
Yes and no.
There are different types of dependencies, and there are different rules for them, but here's an overview of the best practices:
1. For applications, scripts and services (i.e. "executable code"), during development, pin your direct dependencies; ideally to the current major or minor version, depending how much you trust their their authors to follow SemVer. Also make sure you regularly update and retest the versions to make sure you don't miss any critical updates.
You should not explicitly pin your transitive dependencies, i.e. the dependencies of your direct dependencies -- at least unless you know specifically that certain versions will break your app (and even then it is better to provide a range than a single version).
2. For production builds of the above, lock each of your dependencies (including the transitive ones) to specific version and source. It is not really viable to do it by hand, but most packaging tools -- pip, Poetry, PDM, uv... -- will happily do that automatically for you. Unfortunately, Python still doesn't have a standard lock format, so most tools provide their own lock file; the closest thing to a standard we have at the moment is pip's requirements file [0].
Besides pinned versions, a lock file will also include the source where the packages are to be retrieved from (pip's requirements file may omit it, but it's then implicitly assumed to be PyPI); it can (and should, really) also provide hashes for the given packages,…
I was only asking why you say it comes crashing down, when the link cites multiple cases of successful parallel version maintenance with a version that is at least partially calendar-based.
As far as usability, time is a dimension no developer can escape. We develop software on a schedule and we deprecate it on a schedule. CalVer is tremendously semantic at that higher level. The semantics SemVer clings to suffer even under ecosystems which have mandated their use (go, node). The effect has been an over-granularization of APIs themselves.
I'm fine with agreeing to disagree for now. I'm also fine with the case that CalVer is not for everyone and everything. I'll leave it with this, a post seems to articulate a lot of your concerns: https://caremad.io/posts/2016/02/versioning-software/
It was written in 2016 by Donald Stufft, a maintainer of pip. Today, pip has been CalVer for a while and I'm not sure anyone has looked back. Try it sometime, and you might find the same. :)
When I read things like,
https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago.
In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosquatting problems. That's not a coincidence. That's also a strong reason why Java still dominates the enterprise.
If you (the reader) are a PyPi/NPM user, I challenge you to watch this,
https://www.youtube.com/watch?v=pBJafU0p_Nk
and tell me why you shouldn't use a repository manager like Sonatype Nexus which validates package signatures, checks licenses, and does vulnerability scanning.
By itself, package signatures aren't the holy grail? That's not the point. Security is achieved in layers. Too bad they aren't mature enough to understand that.
Which you do not describe, but set that aside: A post that honestly said "we do not like PGP, but here is our alternate plan" would be great. On an actual better solution, I don't think anyone has proposed a good one. Here is the closest I've seen from PyPi (or at least linked from this post as describing their thinking), from 10 years ago:
"Everything is Terrible So What Do We Do?
Bluntly put, I don’t know for sure. This isn’t an already solved problem nor is it an easy to solve one."
https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
What I'll say on PGP is the perfect is the enemy of the good. It's not a tech anyone has much fun using, but in a group setting, used regularly, I have found it can fade into the background at least. I don't want to go any further down the "is PGP good or bad" rabbit hole than that.
But if you have a better solution for package security, please do describe it here.