Travis Oliphant

Original NumPy author here. I have a lot to say on this topic, given that it has literally consumed my life over the past 20 years. You can go here for some thoughts about some of this: http://technicaldiscovery.blogspot.com/ There are several articles there that relate but in particular http://technicaldiscovery.blogspot.com/2012/10/continuum-and-open-source.html and http://technicaldiscovery.blogspot.com/2017/02/numfocus-past-and-future.html

I knew what I was getting into when I wrote NumPy. I knew there was not a clear way to support my family by releasing open source software, and I knew I was risking my academic career.

I did it because I believed in the wider benefit of ideas that can be infinitely shared once created and the need for software infrastructure to be open-source --- especially to empower the brightest minds to create. I did it because others had done it before me and I loved using the tools they created. I hoped I would inspire others to share what they could.

There have been a lot of people who have helped over the years. From employers willing to allow a few hours here and there to go to the project, to community members willing to spend nights and weekends with you making things work, to investors (at Continuum) willing to help you build a business centered on Open Source.

There are many people who are helping to fix the problem. In 2012, I had two ideas as to how to help. Those who know me will not be surprised to learn that I pursued…

Travis Elephant has done very well with Enthought and Continuum Analytics. Before that though, he was an assistant prof at BYU. He's written several times about how he essentially gave up his academic career to continue working on NumPy. For example, he alludes to it in this post he wrote http://technicaldiscovery.blogspot.ca/2012/01/transition-to-continuum.html that says "[my wife] watched me sacrifice my tenure-track position by writing NumPy instead of more papers."

You agree that NumPy has been a huge boon to the scientific community and that BME (his previous field) has reaped the benefits too, right? Let's say NumPy has made minor (~5%) contributions to, say, 500 papers; this is probably a huge underestimate. If he had instead published (say) 25 papers, he would have a thriving academic career and yet his actual impact on the world would be a lot lower. Academia is really bad at rewarding work that has a broad, diffuse impact. In a better world, administrators and funders would recognize that his work on NumPy was sufficiently valuable that he wouldn't have had to choose.

I'm not sure this is the right article to get into "research funding==financing hobbies" debate, but...given that some research is probably going to be funded anyway, wouldn't you rather it be done in an efficient way? Right now, most academic software is a bit of a waste: it's written so that the authors can write a paper about having written it. There's no incentive to produce code that others…

I am not quite sure. Let me be more specific.

Dask requires you to encode computations as a graph explicitly, essentially forcing you to write an AST manually. This means that you are sidestepping the normal language mechanisms to encode program flow, and doing so is essentially incompatible with anything else. Moreover, it does not use numpy arrays - it uses its own internal format that you can convert to a numpy array when needed as explained the overview: http://dask.pydata.org/en/latest/array-overview.html

Numba is another cool project, but - again - it deviates from the mainstream Python. Not every Python function is compilable by Numba, and arithmetic is fixed-size. This means that existing Python code may or may not be compilable by Numba, and even if it works one has to carefully check that arbitrary precision arithmetic is not used. So, it is nice when it works, but is little more than a way to write C-like code with a Pythonish syntax (in which case I greatly prefer Nim, that has actual dispatching on types, generics and so on).

Blaze is in a strange position which I do not fully understand. Apparently it uses Numpy arrays, but at the same time the Numpy author states it should be a Numpy replacement: http://technicaldiscovery.blogspot.it/2012/12/passing-torch-of-numpy-and-moving-on-to.html

Numexpr requires you to write your code as strings, so it is essentially a separate interpreter. It is nice that it is fast, but it does not play with normal Python…

PyPy is an absolutely amazing piece of technologies and the core developers are brilliant people - however, their main use case is not numeric python.

Their framework is (theoretically) able to trace any Python operation, no matter how dynamic, and speed it up. This means that if you want to speed up pure Python code, PyPy is really the only game in town.

The downside is that when scientists use Python, Python is used as a beautiful API on top of very optimized code written in Fortran or C. If you want to do numerically complex code in Python, you are much, much better off using numba. numba is much less ambitious than PyPy - it handles a small subset of Python (basically, NumPy) but it is very, very fast and very efficient at speeding up pure NumPy code. In my experience, 100x speed ups (over pure NumPy code) are not that uncommon.

The founder of continuum (Travis Oliphant) wrote a blog about his technical vision for a Python jit: http://technicaldiscovery.blogspot.it/2012/08/numba-and-llvmpy.html and http://technicaldiscovery.blogspot.it/2012/07/more-pypy-discussions.html). Basically, the continuum team made a big bet that a very efficient JIT that targets only numerical python would be more useful than a generic JIT that can theoretically handle all of Python. For my use case (scientific coding) - numba is far superior.

There's more inertia here than you may be thinking. See this recent HN headliner about the difficulty of getting the NumPy/SciPy community to merely move from CPython to PyPy: http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html

If that's a challenge, good luck getting a lot of NotProgrammers to move to a completely different language.

Besides, if I may be blunt, who in Rubyland cares if these guys are on Python? By and large, these people are on an island. They are optimizing for things you pretty much don't care about, for programming styles you're not using, for libraries you've barely heard of. Trying to entice them off of Python onto Ruby is a waste of time for everybody. The Ruby community would at best get nothing, at worst net pain of supporting a lot of newbies, and the scientists would incur the horrifying overhead of a language shift for the closest thing to a sidegrade available in the set of the top 20-ish language. If they were going to do that they probably shouldn't be going for Ruby, they should probably be going for Haskell or OCaml, both languages for which I could outline actual advantages to the scientists for the pain of the switch (though probably not enough to justify the switch).

Here is the post at Travis Oliphant's blog about numba

http://technicaldiscovery.blogspot.com.es/2012/08/numba-and-llvmpy.html

As it is said there, it's still early software and its "road-map is being defined right now by the people involved in the project". Sure the subset is not well defined now and there are no docs, but hey - let's give the thing a few months.

Haven't tested thoroughly yet, but I think no NumPy calls can be made from inside a numba-compiled function - so this might be the case for other Python modules as well.