The Delta Lake project blog covers the open table format, its connectors, and comparisons with Parquet and other data lake formats.
Commenters call Delta great for easy upserts on Databricks and recommend it over plain Parquet for transactions and update, delete and merge operations.
Can confirm that running Spark at scale is difficult. Not even necessarily talking about scale of data or scale of performance, but organizational scale. Getting dozens or hundreds of engineers aligned around best practices, tooling and local development for Spark is both challenging and extremely rewarding. When you have everyone buy into Spark as not just an execution environment but a programming paradigm, it really unlocks some cool potential. If anyone cares this is how I've found to best get Spark users riding on rails:
* Use a monorepo to "namespace" different projects/teams/whatever. Each namespace has its own build.sbt for Scala jobs and Conda/Pip requirements file for PySpark. This gives you package isolation so that different projects can bump requirements at their own pace. This is crucial in larger organizations where you might have more siloed development or more legacy applications.
* Build each project in the monorepo into a separate Docker image and tag it accordingly with some combination of the branch and namespace.
* Deploy applications onto Kubernetes by invoking the SparkOperator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator), This abstracts away a lot of the hassle of driver/executor configuration and gives you nice out-of-the-box functionality for scraping Spark metrics.
* For local development, use some type of CLI or Makefile to build/run the image locally. This is where the implementation diverges somewhat from using…
1. Data Warehouse
Recommend reading up on Kimball vs Inmon.
Kimball -- centralised storage in a "dimensional model". Uses a star/snowflake schema where the central FACT table (containing the raw transactional data) can be filtered and aggregated based on choices made in dimension tables (edges of the star/snowflake).
demographic account_types product_types
\ / /
\ / /
customer product
\ /
\ /
FACT table (sale $, order quantity, order ID, product ID)
customer, account_types etc are dimensions to filter your low-level transactional data. The schema looks like a snowflake when you add enough dimensions, hence the name.
The FACT table makes "measures" available to the user. Example: Count of Orders. These are based on the values in the FACT table (your big table of IDs that link to dimensions and low-level transactional data).
You can then slice and dice your count of orders by fields in the dimensions.
You could then add Sum of Sale ($) as an additional measure. "Abstract" measures like Average Sale ($) per Order can also be added in the OLTP backend engine.
End users will often be using Excel or Tableau to create their own dashboards / graphs / reports. This pattern makes sense in that case --> user can explore the heavily structured business data according to all the pre-existing business…
Hi, Aleks here, one of the authors, and thank you very much for your comment
We run this stack in production for the last few months, and it has its downsides (I would argue due to the young ecosystem) and upsides, which we try to explain in the blog. We wanted to concentrate more on the concepts and technology change/improvement that allows us to run such a stack and explain how we see the future steps forward.
1. Running a warehouse is not a bad idea, but you must always be careful to separate the storage from the compute to scale. I experienced the limitation of such a system as described in this blog https://delta.io/blog/2022-09-14-why-migrate-lakehouse-delta-lake-tmo-dsna/ -> tough position if your solution is good but not scale.
Ideally run it with external tables in order that data is visible without engine access
Another limitation is the metastore for your tables and metadata, which you usually have per workspace/environment in such a scenario. Databricks' unity catalog is an excellent way to solve it, but it is only compatible with some engines.
2. We do not think that this stack exists to exchange the snowflake or big query but to take a part of the workload away ( data transformation) and let PaaS solutions be good at what they are made for -> user interface and interaction.
> I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it
They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it.
Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) .
1. https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html
2. https://databricks.com/blog/2017/12/01/transparent-autoscaling-of-instance-storage.html
3. https://delta.io/
4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-elastic-block-storage/
Looking at this now.
* Delta Lake supports merge-on-read via deletion vectors: https://delta.io/blog/2023-07-05-deletion-vectors/
* Why doesn't Delta Lake have efficient bulk load? Lots of the biggest datasets in the world are in Delta tables.
* Delta Lake definitely supports compaction: https://delta.io/blog/2023-01-25-delta-lake-small-file-compaction-optimize/
* What does CLI support mean in the context of a Lakehouse storage system? You can open up a Spark shell or Python shell to interface with your Delta table. That's like saying "CSV doesn't have a CLI". I don't get it.
I didn't do a detailed review of the post.
I would suggest to look onto Delta Lake (https://delta.io/) - it's built on top of the Parquet, but has advantages over the plain parquet:
- transactions - you don't get a garbage in your table if your write failed
- supports update/delete/merge operations (in some implementations)
- metadata allows faster discovery of data, for example, if you have a lot of partitions on the cloud storage.
- metadata also allow to support features like data skipping, when you can filter out files that doesn't contain necessary data
- time travel - you can get back to previous versions of the tables