Jake VanderPlas

When people say cpython is slow they are generally pointing to two things

1) The interpreter is _slow_

2) You can't achieve real thread based parallelism

I think in general people have a high level concept of (1) as static vs. dynamic and interpreted vs compiled is an understandable trade off. CPython as an implementation generally gets type casted as slow for its default dynamism which have understandable negative performance implications[1]. However CPython also gives you lots of ergonomic ways to push your program towards the static/compiled end of the spectrum with things like pandas/numpy/numba/extensions. In general using these correctly put you within the ballpark of _faster_ languages, could you write faster assembly by hand? Sure! Is optimizing in another language worth your time? I don't know.

I've never really understood (2) the lack of threading as a problem, multi process parallelism can be accomplished fairly easily if you are CPU bound or projects like uvloop[2] make async tasks fast enough to compete with any web framework out there. Furthermore even though your cpu cycle cost may be levered to a considerable point while operating over hundreds/thousands of servers doing distributed computing right is still hard and developing something like celery/airflow/luigi/dask from scratch is not cheap either. Leaning on CPython's massive ecosystem can massively lower the barrier to entry in a lot of big problems.

I think there are plenty of examples of re-writes…

This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1].

[1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/

But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type.

[2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/

So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching).

But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy…

I... am not sure that I understand what you mean ?

That after shuffling you know less about the stack of cards (that used to be at least partially revealed) is a fact that our model must follow or fail at being relevant.

It's been hard to find out more information about this, but I did find some :

https://www.preposterousuniverse.com/blog/2015/08/11/the-bayesian-second-law-of-thermodynamics/

Same thing : "And the Bayesian Second Law (BSL) tells us that this lack of knowledge — the amount we would learn on average by being told the exact state of the system, given that we were using the un-updated distribution — is always larger at the end of the experiment than at the beginning (up to corrections because the system may be emitting heat)."

Though I also did find another interesting thing :

http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/

"in a strict frequentist view, it is meaningless to talk about the probability of the true flux of the star: the true flux is (by definition) a single fixed value, and to talk about a frequency distribution for a fixed value is nonsense"

But that's pretty much the case in statistical physics ! A macrostate is actually NOT a state (=microstate), but a probability distribution !

"For Bayesians, probabilities are fundamentally related to our own knowledge about an event. This means, for example, that in a Bayesian view, we can meaningfully talk about the probability that the true flux of a star…

A more recent article addressing this essay from the same author:

https://jakevdp.github.io/blog/2014/08/22/hacking-academia/

The tl;dr summary from this second article referencing the first:

a quick summary is this: scientific research in many
disciplines is becoming more and more dependent on the
careful analysis of large datasets. This analysis requires
a skill-set as broad as it is deep: scientists must be
experts not only in their own domain, but in statistics,
computing, algorithm building, and software design as
well. Many researchers are working hard to attain these
skills; the problem is that academia's reward structure is
not well-poised to reward the value of this type of work.
In short, time spent developing high-quality reusable
software tools translates to less time writing and
publishing, which under the current system translates to
little hope for academic career advancement.

I think the HN title is somewhat misleading, the central thesis from the original article is stated as:

the skills required to be a successful scientific
researcher are increasingly indistinguishable from the
skills required to be successful in industry

This is probably a stunningly awesome outcome - it means that industry has a place for advanced degree holders who will not find a classic academic position. In the linked essay above, from the same author:

the number of PhDs granted each year far exceeds the
number of academic…

Quite a bit has changed in the last three and a half years when it comes to Python packaging. What specifically did you want to highlight here, and why?

Anyway, some responses:

> there’s one thing that irritates me every single time I revisit the page and that is the misleading recommendation of their own tool pipenv.... PyPA recommends pipenv as the standard tool for dependency management... PyPA still advertises pipenv all over the place and only mentions poetry a couple of times, although poetry seems to be the more mature product.

This looks very different now, but I don't think the recommendation was "misleading". The archived page didn't say that the tool was "the standard", just a recommendation. Meanwhile, Poetry lagged behind for years in implementing standards. Pipenv was also focused only on dependency management (which basically == creating venvs + invoking Pip + updating lockfiles), whereas Poetry is designed as a complete workflow tool (== dependency management + build frontend + build backend + package uploader + project metadata management + a bunch of integration + probably stuff I forgot).

> You would expect exactly one distribution for Python packages, but here in Python land, we have several ones.

Conda came into existence because the scientific Python community needed to solve packaging problems right now (around 2012 - see…

There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a different design in how to build a package and also how to bundle it. It specifically handles compiled code much better, because in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. While in a certain sense of reproducibility can be achieved with PyPI packages, my understanding is that, eg with manylinux, it takes a hope for the best approach and can break in exotic cases, which should never happen with conda packages.

> The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own"

From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/