Matthew Rocklin, creator of Dask, writes about Python data tooling, benchmarks, software practice, and AI.
Commenters like the blog and cite his posts on biased benchmarks, writing dumb code, and Dask compared with Spark.
I work on Dask now and the creator summarizes his preferred code style nicely in this post: https://matthewrocklin.com/blog/work/2018/01/27/write-dumb-code. I like working on simple codebases like this.
The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-datasets/. Don't think the lib is really usable. Typelevel stuff is not what I'm referring to when I say better Python. I'll try to make another blog to clarify.
I think Scala is a better language than Python. But for some reason it just seems like Scala is full of mind bending code that I can't understand. Full disclosure: I am way less smart than lots of the Scala ppl I've worked with.
Hi Peter, good to run into you again :)
I agree, formulating open benchmarks is great work, and there's nothing suspicious about performing well on benchmarks you write. It's just worth keeping in mind: https://matthewrocklin.com/blog/work/2017/03/09/biased-benchmarks
My initial remarks were a little careless and overly provocative! I don't doubt that there are cases where a true "array database" provides value. I do think the use-cases are less clear for arrays than they are for tabular data, because the users of arrays tend to be more sophisticated.
Distributed [1] is very new and seems to have similar core architectural goals as TensorFlow. But perhaps I'm being too politically correct: both make use of very course-grain parallelism relative to a typical distributed-memory linear algebra library (e.g., the current communication mechanisms of both are likely to be too course-grain to efficiently support distributed dense matrix inversion or eigensolvers; not that this is likely to be a design goal of either).
[1] http://matthewrocklin.com/blog/work/2015/06/23/Distributed/
FWIW the scientific computing community (who often deal with petabytes of geodata) has been thinking of ideas like this for a while, e.g. techniques around file formats that are easy to lazily and partially parse, (ab)using FUSE to do partial reads using http RANGE requests, some combination thereof, etc:
http://matthewrocklin.com/blog/work/2018/02/06/hdf-in-the-cloud
This https://docs.dask.org/en/stable/spark.html notes
"However, Dask is able to easily represent far more complex algorithms and expose the creation of these algorithms to normal users [compared to spark]" linking to:
http://matthewrocklin.com/blog/work/2015/06/26/Complex-Graphs
BUT WHY
Your performance is going to be complete and utter crap because you're paying for serialization on every single data element.
Dask is higher performance and more pythonic: http://matthewrocklin.com/blog/work/2016/02/22/dask-distributed-part-2