Wes McKinney, creator of pandas and co-creator of Apache Arrow, writes about data tools, columnar formats and software development with AI agents.
Commenters treat him as an authority: his book is called the standard, and his pandas memory rule of thumb of 5 to 10 times data size is quoted.
I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data.
Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/
I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient.
Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-arrow-native-future-cbafe27b6eb7
Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin.
The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).
The first edition of this book was a game changer. I still have my hardcopy paperback but haven’t turned to it in years. It’s probably largely out of date but I look at it fondly. Glad to see the material updated with new APIs and Python 3.10. The open access is very nice because it can easily be used for teaching and reference.
I would like to see some of Wes’ cautionary tales included into the book. Pandas feels so magical at times that you can uses it to solve anything. That can create problems. Ideas like this should be some appendix to the book.
https://wesmckinney.com/blog/apache-arrow-pandas-internals/
Lots of tools in the data space. It feels like choosing JS web front ends. I use pandas quite a lot but less frequently, as my data has gotten larger I turn to spark more. I’m also impressed by the work DuckDB is doing.
I went the same route but ended up adopting feather from the makers of Python's Pandas and R's tidyverse, Wes McKinney and Hadley Wickham. https://blog.rstudio.com/2016/03/29/feather/
> SUMMARY: Feather's good performance is a side effect of its design, but the primary goal of the project is to have a common memory layout (Apache Arrow) and metadata (type information) for use in multiple programming languages. http://wesmckinney.com/blog/feather-its-the-metadata/
The approach is data frames on disk storage.
> data frames are lists of named, equal-length columns, which can be numeric, boolean, and date-and-time, categorical (_factors), or _string. Every column can have missing values.
This fits the needs of all of my CSV usage.
Your first paragraph makes perfect sense! I was nodding along. But then your concluding sentence was a bit of a record scratch for me. This all worked as intended! You knew what the project was about - "data frames" - and what might make it attractive to you - the performance claims - and then you went and followed exactly the right path to get the context you needed to understand what's going on with it. It's a big topic that you were able to spin up on to a basic level in 2-3 hours, by pulling on strings starting at this landing page. This is a very successful outcome.
I'd also recommend this book: https://wesmckinney.com/book/. It's not about polars, but you'd be able to transfer its ideas to polars easily once you read it.
> large datasets on disk
I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414
Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that could further optimize for various data.frame use cases. As an added bonus, the same libraries would be very portable between different languages--even languages that don't currently have something like dataframes.
EDIT: What I don't know about is memory efficiency. Perhaps SQLite isn't, but I'd not bet against?
I could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library.
There's a lot of related discussion in this post on his blog.
https://wesmckinney.com/blog/apache-arrow-pandas-internals/