Gregory Szorc

The thing that put me off of Rye is that it strongly encourages you to use non-standard Python builds.

From their philosophy page: https://rye.astral.sh/philosophy/

> No Python Binary Distributions: CPython builds from python.org are completely inadequate. On some platforms you only get an .msi installer, on some you literally only get tarballs. The various Python distributions that became popular over the years are diverging greatly and cause all kinds of nonsense downstream. This is why this Project uses the indygreg standalone builds. I hope that with time someone will start distributing well maintained and reliable Python builds to replace the mess we are dealing with today.

And here is info about those particular indygreg builds.

https://gregoryszorc.com/docs/python-build-standalone/main/

It is, however, possible to choose a different Python.

https://rye.astral.sh/guide/toolchains/

I've never really experienced the problem they are describing. Any official Python build I've gotten from python.org has worked. Every normal old Python I've gotten from my distribution's package manager has worked. Every Python included with an official Docker container image has worked.

I'm sure their special builds will also work just fine, but their non-standardness gives me pause. There's even a list of behavior quirks. Why are we adding quirks to something that has no problems? And the fact that the rye philosophy seems so intent on using them, turns me off from it compared to…

Another great article on this theme:
https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/

It is frustrating that we seem to be evolving towards the nesting of one build system inside another, such as "go build" inside "docker build". (One of the projects I work on uses janky, docker compose, docker build, Go build, Make, Rake, and Bundle.) This approach is opaque, coarse-grained, inefficient, unnecessarily sequential, and doesn't materialize "the" build graph as a single entity from which we can identify our software artifacts and their relationships and derive and automate all our workflows.

Bazel addresses these problems with a single, canonical, first-class graph, and many projects that use Bazel achieve close-to-ideal build efficiency, scalability, and reproducibility, as well as a good foundation for other workflows. But switching to Bazel is a large undertaking for a project that has already become entangled in the kind of situation described above. Maintaining hand-written or generated BUILD files for dependencies can be a burden. (I led the design of Blaze in 2006, at which time Google was already using BUILD files to declare software artifacts and dependencies in its more primitive predecessor system; we simply could not have achieved Blaze even at Google's then scale otherwise.) I often dream of a future world in which it is expected that compiler writers will provide a formal definition of the toolchain's workflow in a form that…

The Python 3 upgrade process for many projects was incredibly painful. "Mercurial’s journey to and reflections on Python 3" should be required reading for anybody with rose-tinted glasses of the migration.

https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/

There was, of course, a Hacker News thread discussing the article, and a fair few people decided to blame the Mercurial developers for handling the migration inelegantly. Because that's how you win over an audience of developers - reassuring them that if Python has a backwards-compatibility break, Python fans will go out of your way to try and blame you for writing bad code. And not, perhaps, the fact that Python was missing things like a u string prefix and % bytestring formatting until 3.3 (2012) and 3.5 (2015!!!) respectively.

If I sound peeved, I really loved Python in the 2.x days, and the way the 3.x transition was handled broke my heart and prevented me from using the language for pretty much an entire decade. There are lessons to be learned from the transition, but not if we ignore the real problems that the transition caused. More importantly, we need to recognize that Python is not the Python we know today because of how "well" the transition was handled, but because Numpy and Matplotlib swooped in and gave Python new niches to excel in at just the right time.

This is great. But please note, some of the graphs presented use a common statistics cheat of logarithmic scale along one or more of the axis.

Many times, lz4 level 1 isn't just 25% faster [0] in Compression vs. say zlib level 1 (as it appears visually), it's actually 6 times (6x, or 600%) faster in terms of MB/s (~600 MB/s vs. ~100 MB/s)!

And it's not just 5% or so faster than zstd in Decompression (visually) [1], it's about twice as fast (~2000 MB/s vs. ~1000 MB/s).

Yes. It does not compress as well. But ~3.2:1 compression ratio best case for lz4 (listed as ~3.2 in the chart) compresses almost as quickly as the 6:1 zstd [0] but will then decompress twice as fast as zstd at any level [1]. Having a high-compression is a case I could (almost) not care about. If you're youtube/google or facebook/instagram, you might care (streaming, photos, static assets all at enormous scales).

For me, however, the above results mean far less CPU burden on the client with a twice the burden on the network I/O. If you're concerned about battery, loading screens, or download dialogs, I'd still pick lz4. Just turn the AC up a half notch in my AWS DC.

[0] http://gregoryszorc.com/images/compression-bundle-common.png

[1] http://gregoryszorc.com/images/decompression-bundle-output.png

As the sibling comment says, horizontal scrolling would be an improvement over the status quo. Pinch to zoom would be another nice win.

> Can you tell me more about what you use this feed for? You're talking about the one on the github.com home page?

Yes, that's the one I'm talking about. I use it to see who's starring repos I'm involved in, who's following me, and what the developers I follow are starring (the feed also has stuff about what people are pushing to repos I watch, but that is not interesting to me because I usually keep track of it by following PRs or issues). Just keeping track of what's going on in the world.

Also, now that I have your attention, please please read https://gregoryszorc.com/blog/2020/01/07/problems-with-pull-requests-and-how-to-fix-them/ and make life better for those experienced engineers who do per-commit reviews.

Finally, I found the feedback handling from the betas somewhat discouraging. At least when I tried it, it just emailed off and there was no follow-up at all. If that process was a bit more conducive, I might have given you this feedback sooner.

There is one thing you can't with most algorithms: prallelize decompression. That's because most compression algorithms use sliding windows to remove repetitive sections.

And decompression speed also drops as compression ratio increases.

If you transfer over say a 1GBit link then transfer speed is likely the bottleneck as zstd decompression can reach >200MB/s. However if you have a 10GBit link then you are CPU bound on decompression. See e.g. decompression speed at [1].

Bzip2 is not window but block based (level 1 == 100kb blocks, 9 == 900kb blocks iirc). This means that, given enough cores, both compression and decompression can parallelize. At something like 10-20MB/s per core. So somwhere >10 cores you will start to outperform zstd.

Granted, that's a very very corner case. But one you might hit with servers. That's how I learned about it. But so far I've converged on zstd for everything. It is usually not worth the hassle to squeeze these last performance bits out.

[1] https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/