SF Python
4 HN recommenders · recommended in 7 months, 2018–2023
1:13:53 · 161K views · Aug 31, 2017
Raymond Hettinger's PyBay 2017 keynote on concurrency in Python. It covers threads, processes, asyncio, locks and the GIL, with live shell-based examples.
Recommended as a clear, charismatic starting point for understanding Python concurrency and the GIL.
7 comments from Hacker News
I've just left the question open, but the best learning material for me on this was this lecture. Other points down the list, I had to figure out just by trial and error. Raymond Hettinger, Keynote on Concurrency, PyBay 2017 this video
I've started watching Raymond Hettinger's talks lately. He's a core dev on Python. I love them, the guy is charismatic, very smart and his Python knowledge constantly impresses me. [LINK] My favourite: this video
For anyone looking for something on the same topic but a little bit lighter / more practical, I really enjoyed this talk[1] by Raymond Hettinger at PyBay 2017, where he discussed the GIL as well. [1] this video
It is because only one thread at a time holds the lock in order to avoid race conditions. The keynote[1] by Raymond Hettinger from PyBay '17 will be a great place to start if you are new to this. [1] this video
To say concurrency is broken in Python because of the GIL seems to be reiterated adage without actual wisdom behind it.[1] Are you writing an IO-bound task? Then asyncio[2] (even if just viewed as an interface: e.g., you can make it even more performant by using uvloop[3]) is a great way to achieve performant concurrency. As Raymond Hettinger likes to point out, even if you throw away the GIL, you will have to implement locks on shared resources (unless you're writing embarrassingly parallel codes--in which case either you can invoke them separately or use the multiprocessing module to great effect, or see how I do it easily below), and by the time you get all the locks implemented in your GIL-less world (locks which are error-prone), you've given up the performance you thought you'd get due to throwing away the GIL. Are you writing a CPU-bound task? I've only encountered this in scientific computing, as I'm a scientist, and for this (as another commenter mentions) I either use numpy[4] (strictly within a numpy call, the GIL is not held) or numba[5] (by default the entirety of your numba-fied functions do not hold the GIL). Therefore, I can either use Python threads or asyncio or Numba's built-in parallelism[6,7] (summary: some numpy routines for working with arrays of values are auto-parallized if called within a numba-fied function, and you can explicitly parallelize your own loops via `for i in numba.prange(N):`; in any case, then decorate the function with `@numba.njit(pa