Tidyverse Blog

True, the default semantics of R's data structures are somewhat arcane (of course as they're based on S [1] from the 70's). And the current support for e.g. 64bit integers leaves something to be desired.

But behind the scenes, R is just a lisp with some data structures that are adapted to statistics and data science.

All base data structures are by default immutable. And e.g. the vector type is extremely performant as it's just a thinly wrapped C Array. In Python you need to reach for Numpy for anything similar, and you do feel some pain when converting between native python types and Numpy types for various functions which support one or the other.

The data frame is immensely powerful. And has excellent performance characteristics as it's built upon vectors. A list of objects, like you'd make in python is just a lot slower and more unwieldy to deal with. And much harder to make generalizable functions upon.

Hadley Wickham's Tidyverse[2] is exactly an attempt to hide away the arcane details and create a modern, coherent and consistent language on top of R, keeping the power of all the great statistics R libraries. The fact that R behind the scenes is a Lisp, with support for macros, makes this possible. For doing data-transformations and statistics, I can't think of anything currently as powerful as CRAN + Tidyverse.

[1] https://en.wikipedia.org/wiki/S_(programming_language)

[2] https://www.tidyverse.org/

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files).

I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the power of `broom`,fitting simple models to the data and working with the results is really nice. Add to that that `ggplot` (+ any sane styling defaults like `cowplot`) is the fastest way to iterate on data visualizations that I've ever found. "R for Data Science" [1] is great free resource for getting started.

Snakemake [2] is a pipeline tool that submits steps of the pipeline to a cluster and handles waiting for steps to finish before submitting dependent steps. As a result, my pipelines have very little boilerplate, they are self documented, and the cluster is abstracted away so the same pipeline can work on a cluster or a laptop.

[0] https://www.tidyverse.org/

[1] http://r4ds.had.co.nz/

[2] http://snakemake.readthedocs.io/en/stable/

Data frames in R give you the relational model with a Turing complete language. I think of it as a better SQL, at least for analytics (as opposed to transaction processing).

Besides its data structures, R is surprisingly similar to JavaScript -- it's has Scheme-like semantics but C-like syntax.

For certain problems, it's a pleasure to program in. You don't have to go through "objects" to get to your tables; you just have tables!

The examples here might give you a flavor:

https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html

R itself has a lot of flaws, but the tidyverse [1] is collection of packages that provide a more consistent interface.

FWIW I agree that more mainstream languages should have tables natively. It is an odd omission. Somehow they got coupled to storage, but they're also useful for computation.

[1] https://www.tidyverse.org/

Under the hood, the tidyverse is comprised of many packages, some very large and sprawling, like ggplot, and others that do one single thing (readr). Here’s a list: https://www.tidyverse.org/packages/

Individual packages can be used in isolation, if that’s what you need. However, they complement each other nicely, in that they share data representations and similar conventions while handling different facets of data analysis. Thus, people often import the whole thing in one shot as the tidyverse and go from there. For example, you might read the data in with readr, clean it up with tidyr or dplyr, analyze it with tidymodels, and plot the results with ggplot.

The python equivalent would be a hypothetical meta-package that tightly integrates pandas, seaborn, and sklearn.

Trying to find a good reference for you. There’s this from the author of dplyr:

https://news.ycombinator.com/item?id=30067406

And here in the docs:

https://dplyr.tidyverse.org/reference/dplyr_by.html

And maybe also:

https://www.tidyverse.org/blog/2023/02/dplyr-1-1-0-per-operation-grouping/

I guess part of it is that there’s some ‘non-locality’ in the pipeline where the grouping could be relatively distant from the operation acting on the grouped data. Similarly, you get to worry about eg grouping data that is already grouped.

I quite like the prql solution which is to have a ‘structured grouping’ where you have to delimit the pipeline that operates on grouped data, but maybe it can still lead to bad edits for complex queries.

There's nothing about RStudio that encourages big single files or writing huge unstructured scripts. RStudio is a pretty good IDE, and R is a highly expressive functional-first [0] language. R was heavily influenced by Scheme, and has its own powerful metaprogramming [1] system - which is used to great effect in Tidyverse[2] libraries to make APIs that are nicer and convenient than anything reasonably practical in Python.

The problem with a lot of end-user R code is that it is written by statisticians, not programmers. They'd write the same garbage and huge scripts in Python (trust me, I know).

[0] http://adv-r.had.co.nz/Functional-programming.html

[1] https://adv-r.hadley.nz/metaprogramming.html

[2] https://www.tidyverse.org/