HPy Blog

I've battled with Python's packaging systems and other tooling and performance problems for more than a decade. Every couple of years some new thing comes along promising to fix package management, but it always fails spectacularly on some important, mainstream package or use case. I suspect a lot of this complexity comes down to the significant degree in which the Python ecosystem depends on C: not only do we need to package Python programs but C programs as well, and some people think we should ship C source code and build on the target machine and others think we should ship pre-built C binaries and dynamically link against them on the target machine thus Python supports both mechanisms; however, both are perilous in the general case.

And interestingly, Python depends so hard on C because CPython is very slow, and CPython is very slow because the C-extension API is virtually the whole of the CPython interpreter (many interesting optimizations to the interpreter would break compatibility in the ecosystem). So it certainly feels like all of Python's major problems come down to this decision to lean hard on C and specifically to make so much of the interpreter available for the C-extensions.

The way out as far as I can tell is to define some narrower API surface (e.g., https://hpyproject.org), get the ecosystem to consolidate around that, deprecate the old API surface, and then make the requisite breaking optimizations such that the ecosystem can feasibly do more in native…

Hey!

First of all, cool to see some activity on this front!

I’ve written a fair share of pure CPython bindings and regularly post about implementing them with minimal overhead (<https://ashvardanian.com/posts/discount-on-keyword-arguments-in-python/>) and would love to share a few recommendations, questions, and concerns :)

Just a suggestion to help you grow—I'd restructure the landing page (<https://hpyproject.org/>) and the README of the repo (<https://github.com/hpyproject/hpy>). It could benefit from some examples to clarify the "Nicer API" bullet point. Maybe these could be taken from the API documentation page (<https://docs.hpyproject.org/en/latest/api.html>). The page could also be more convincing with some supporting stats in favor of PyPy, GraalPython, and other Python runtimes. A reader like me might not be sure if they have enough usage and are stable enough.

Avoiding singletons and having encapsulated context objects like `HPyContext` is definitely a great thing to have, especially in the multi-threaded Python future or in complex environments with multiple sub-interpreters. But this doesn't really solve the problem if, under the hood, the `HPyContext` still redirects to CPython's singleton.

I've also looked at the linked benchmarks (<https://pypy.org/posts/2019/12/hpy-kick-off-sprint-report-1840829336092490938.html>). They are dated from 2019, five years ago, and already mention CPython's `METH_FASTCALL` fast calling convention, but it seems like they are…

> numpy (and swig) predates Unladen Swallow

Sure, but that doesn’t contradict what I said. Numpy is surprisingly old, but in 2010 (my impression was that) supporting it was in no way table stakes for an implementation of Python. Hell, Jython was still occasionally taken seriously then, and Microsoft still pretended they cared about IronPython.

> I would suggest the C API is just as much a part of Python as the standard library

As things stand, yes, no question.

The problem is, it was never designed that way. In particular, it cannot (or at least is not designed to) support any other memory management strategy but CPython’s, thus efforts like HPy[1]. For example, while the language docs admonish you not to depend on __del__ running at any particular moment, if we consider the Python/C API a part of the language, then the language has to behave as though it uses naïve reference counting and occasionally runs a cycle collector. And that’s the easy part—think about how deeply the GIL is rooted there.

That is not good design for something that’s part of the language, as it effectively is. It was never intended to be and never received commensurate amounts of care. It just turned out that way—and ain’t that a sad thing to admit.

[1] http://hpyproject.org/

> I think one could radically change the way Python objects work internally, and have the C foreign function interface (FFI) wrap every object passed to a C extension in an API/ABI-preserving facade

HPy is building an API abstraction layer which is designed to be used with both the CPython API and JITs. However, IIUC they are not proposing any changes to CPython itself, but rather to provide a smaller API surface and fewer JIT impedance mismatches when extensions are built against something other than CPython. The lead developer is a longtime PyPy developer.

https://hpyproject.org/ (https://github.com/hpyproject/hpy)

pypy IS compatible with C extensions, via what we call "cpyext": https://doc.pypy.org/en/latest/faq.html#do-c-extension-modules-work-with-pypy

The problem of cpyext is that it's super slow, for good reasons: https://pypy.org/posts/2018/09/inside-cpyext-why-emulating-cpython-c-8083064623681286567.html

There are efforts to create a new C API which is more friendly to alternative implementations (including CPython itself, when they want to change how they do things internally): https://hpyproject.org/ https://github.com/py-ni

Absolutely everything in CPython is a PyObject, and that can’t be changed without breaking the C API. A PyObject contains (among other things) a type pointer, a reference count, and a data field; none of these things can be changed without (again) breaking the C API.

There have definitely been attempts to modernize; the HPy project (https://hpyproject.org/), for instance, moves towards a handle-oriented API that keeps implementation details private and thus enables certain optimizations.