Dustin Ingram, a Python Package Index maintainer who writes and speaks about PyPI infrastructure and how it works.
HN links it as a source on how PyPI is run, especially its 2021 infrastructure post about Fastly donating bandwidth worth millions per month.
>Running our test suite 2-3 times over the span of a couple hours would trigger the rate limit…
Not to single you out, but not too hard to imagine why companies start enabling rate limiting. I oftentimes wonder the obscene amount of bandwidth being consumed by this developer automation. This page[0] indicates that PyPi serves 900 terabytes daily.
Seems obvious we need more turn-key solutions that enable caching for all of the popular package distributions (PyPi, NPM, Docker, etc). Too hard to configure and everyone will default to the path of least resistance (ie hammering the origin servers).
[0] https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/
There are 2 members in the disposable-email-domains organization on GitHub.[1]
You (martenson) are one of them. The other member (di), who is a "core maintainer" of the project according to the README changelog,[2] is also a member of the Google organization on GitHub.[3] di describes himself as part of the "@google open source security team"[3] and the website linked from his profile says that he is "a Developer Advocate at Google".[4]
[1] https://github.com/orgs/disposable-email-domains/people
[2] https://github.com/disposable-email-domains/disposable-email-domains/tree/6354e34bd18e6b2881e381523e40ce2e6e5a7c1c#changelog
[3] https://github.com/di
[4] https://dustingram.com/
Taking PyPI as a central place of packages, it is known that their bandwidth bill would be $1.8+M per month (https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/) were it not for Fastly giving them a 100% discount.
Are there any reliable decentralized package distribution systems operating at within 2 orders of magnitude of that scale? How do they handle administrative issues such as malicious packages or name squatting? Standards updates? Enforcement of correct metadata? And all the other common things package indexes need to handle.
I'm clearly skeptical, but would be very interested in any real world success stories.
Some think it is. Excerpt from this talk by Dustin Ingram:
https://dustingram.com/talks/2018/10/23/inside-the-cheeseshop/ (halfway through)
PyPI is sometimes called the Cheeseshop.
This is a reference to this Monty Python skit, where a man goes into a cheeseshop which has no cheese for sale.
The joke is that when PyPI was first created, there was nothing in it. Hence, "The Cheeseshop".
But it does have stuff in it now, so it's not really even appropriate to call it that anymore.
For its part, PyPI is apparently funded by the Python Software Foundation and run by volunteer admins, with the bulk of its data served gratis by Fastly. This is based on a blog post about its infrastructure from 2021 [1], which also says that the list price for the throughput they're doing is $1.8 million per month.
[1] https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/
The official PyPI packkages are hosted on Fastly. I believe they donate the infrastructure costs through the Python Software Foundation, as the bandwidth is over 300TB a day, and would cost about 1.8 million dollars a month.
Source: https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/