Coiled Blog

We use dask heavily, along with rest of the pydata ecosystem. I guess we are in the 'sweet spot' where data doesn't fit memory, to begin with, but once we perform any filtering and aggregations, switch over to pandas. That's exactly what dask recommends too. Our datasets don't exceed 100GB right now.

Also note, dask clearly acknowledges challenges dealing with data in the terabyte range https://coiled.io/blog/dask-as-a-spark-replacement/

Most of our use-cases right now involve using multiple cores of a big instance, than resorting to cluster computing.

With spark, there is additional/steep learning curve, complexities of dealing with cluster computing. And Spark-ML is not well known. With dask/pandas it's easy enough to feed scikit-learn and/or bring in dask-ml, just a pip install, and you can scale well known sklearn modules effectively.

I think in the end, it's about keeping things simple. As others said, if you are already invested big in Spark/Scala/Hadoop, that may make sense for you. For non-CS folks, this will be a challenge.

As for vaex, it's very interesting. One issue is that it seems to be able to want hdf5 and doesn't want to work with parquet. And it's API is not fully compatible with pandas.

Ray/Modin: played with it a bit and maybe it's a bit too new for enterprise uses and may be more geared for ML workloads. That's my take anyway and it may have progressed substantially, already.

Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are.

Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed (https://blog.jetbrains.com/blog/2020/02/25/update-on-big-data-tools-plugin-spark-hdfs-parquet-and-more/).

Parquet column pruning & predicate pushdown filtering allow for great query performance improvements, see this blog post I wrote for benchmarks: https://coiled.io/blog/parquet-file-column-pruning-predicate-pushdown/

Delta Lake (Parquet files + a transaction log) makes it quite pleasant to manage a large lake of data stored in Parquet. There are tons of other features Delta allows for like time travel, schema enforcement, schema evolution, etc.

Coiled | Remote-first | Full-time | Cloud Infrastructure & Product Engineers | https://coiled.io/jobs/

Coiled connects data scientists and researchers to distributed infrastructure using the Python data science stack (Numpy, Pandas, Scikit-Learn, etc) and Dask, a popular open source library for parallel analytics. We come from the open source Python community, and are long-time maintainers of this stack in general and Dask in particular.

We’re building a product that enables organizations and enterprises to collaborate better using Dask by solving the setup, scaling and security challenges of the standard open source Python stack.

- Cloud Infrastructure Engineer -- https://coiled.io/jobs/#cloudinfrastructureengineer
- Product Engineer -- https://coiled.io/jobs/#productengineer

To apply drop us a line at jobs@coiled.io :-)

Have you taken a look at the "Zero to JupyterHub with Kubernetes" guide? https://zero-to-jupyterhub.readthedocs.io/en/latest/index.html

Alternatively, you could check out the QHub project from Quansight. https://qhub.dev/

I think these are less relevant to you since you already have in-house k8s resources, but Coiled and Saturn Cloud offer services that might be interesting to other people with similar goals but who don't have a kubernetes cluster already.

https://coiled.io
https://www.saturncloud.io/

Not OP, but in my experience:

Jupiter Intelligence (https://jupiterintel.com/)

Descartes Labs (https://descarteslabs.com/)

Microsoft Planetary Computer
(https://planetarycomputer.microsoft.com/)

Coiled (managed Dask - python HPC) (https://www.coiled.io/)

CarbonPlan (https://carbonplan.org/)

Salo Sciences (acquired by Planet, satellite imagery company) (https://salo.ai/)

lots of others