Dremio Blog

I think the reason this is discussed now is because yesterday I tried to re-submit the Apache Arrow article. Here's what I wrote: https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&oldid=931353298
It was rejected / reverted 10 minutes later by a Wikipedia editor.
The blog post from Justin was in July 2019 (https://www.dremio.com/why-apache-arrow-wikipedia/)

There's many interesting and good points in the discussion here, thank you!

To add my 2 cents:

- Apache Arrow is notable, deserves a Wikipedia page. It might not have been when someone first tried to create a Wikipedia page for it in 2017 (see https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&action=history), but in the three years since it has become a major project, see e.g. https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces46 Notability is clearly subjective, depends on what the author and reviewer find interesting. In the variant I submitted yesterday I tried to make it clear why it's notable - Apache arrow is a standard format that connects different languages, runtimes, data systems, communities, e.g. the Python and Java data communities. See e.g. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ - Apache Arrow is to my knowledge partly the brainchild of Wes McKinney, creator of pandas, it's his attempt (looking strongly like success) to resolve a major issue in data science.
- I think it's a good point Justin made at…

Dremio | Full Stack and System Engineers | Santa Clara, CA | Onsite (once shelter-in-place ends) | https://www.dremio.com/

We are looking for Full Stack and System engineers. Our stack is Java and React and open source our core technologies (such as Apache Arrow).

Dremio is the Data Lake Engine company. Our mission is to reshape the world of analytics to deliver on the promise of data with a fundamentally new architecture, purpose-built for the exploding trend towards cloud data lake storage such as AWS S3 and Microsoft ADLS. We dramatically reduce and even eliminate the need for the complex and expensive workarounds that have been in use for decades, such as data warehouses (whether on-premise or cloud-native), structural data prep, ETL, cubes, and extracts. We do this by enabling lightning-fast queries directly against data lake storage, combined with full self-service for data users and full governance and control for IT. The results for enterprises are extremely compelling: 100X faster time to insight; 10X greater efficiency; zero data copies; and game-changing simplicity. And equally compelling is the market opportunity for Dremio, as we are well on our way to disrupting a $25BN+ market.

Find out more at https://www.dremio.com/careers/

Nice article, thanks!

It would be great if DuckDB handled this itself, but it seems like to be competitive with Athena on really massive datasets, you need to have a metadata layer that is used to figure out which parquet files in S3 DuckDB actually needs to query and then potentially run those in parallel. This seems to be the architecture of Puffin (which I haven't personally tried using yet).

[1] https://www.boilingdata.com/
[2] https://boilingdata.medium.com/lightning-fast-aggregations-by-distributing-duckdb-across-aws-lambda-functions-e4775931ab04
[3] https://github.com/sutoiku/puffin

One possible thing to look into would be whether this dataset is partitioned too much. My understanding is that the recommended file size for individual parquet files is 512MB to 1GB, whereas here they are 50MB. It would be interesting to see the impact of the partitioning strategy on these benchmarks.

[4] https://parquet.apache.org/docs/file-format/configurations/
[5] https://www.dremio.com/blog/tuning-parquet/

Interesting project!
I only got to know about Apache Calcite a few months/weeks ago, but it seems like it's becoming a de facto way of supporting SQL on analytics products. I can think of Dremio [1] as an example.

From an implementation perspective, I wonder how AthenaX will compare to the Apache Beam SQL DSL [2] (I haven't tried it out, still learning DoFn, ParDo, etc.). Beam is also leveraging Calcite, with the partial difference that it runs* on Flink amongst others.

I think it's safe to say that the streaming analytics ecosystem remains interesting and exciting! Hope I'll be able to try out AthenaX, congratulations on the release.

[1] https://www.dremio.com
[2] https://github.com/apache/beam/tree/DSL_SQL