lakeFS's company blog covers data versioning, data lake tooling, table formats, and data engineering trends.
HN commenters mention lakeFS as an option for Git-like versioning over S3-style object storage; they give no opinion on the blog itself.
There's a lot of factions doing git backed or git like systems out there already. LogRocket has a round-up of 9 different CMS, such as the excellent excellent Tina CMS. The #GitOps is incredibly well loved & respected for all manner of systems & infrastructure, with projects like ArgoCD being immensely popular. https://blog.logrocket.com/9-best-git-based-cms-platforms/ https://tina.io/ https://argoproj.github.io/cd/
There's countless git-like data systems. Dolt DB, LakeFS, and notably Iceberg has branching. https://github.com/dolthub/dolt https://lakefs.io/ https://iceberg.apache.org/docs/latest/branching/
My experience is the pushback that git is too complex, git is for programmers is enormously fast to follow most suggestions on using git. I remain enormously skeptical. There was a zeitgeist floating around a week or two ago about the worst thing you can do is to make decisions on the assumption that people are stupid, and git is a place where I absolutely believe we can outshine the rain, and will wonder why we ever let such doubt guide us away from such an amazing set of capabilities with such amazing tooling. We have such a start already with git tools! Once we try, we can keep humanizing this!!
My pet theory is that much of the weakness of git for everyone is that we do a terrible job of expressing data on the filesystem. It's popular & so reasonable to be able to copy a document to someone, but in my mind, ideally a document is maybe a file tree of sections or…
saltcured, find these comments super insightful!
> Yeah, there's a lot of hidden magic/assumptions in having a "writable snapshot of a specific version" of production data.
That's absolutely a huge assumption. This technology has been a game changer for us: https://lakefs.io/
> It becomes a headache when there is too much contention to use these sandboxes, or too much manual effort to reset them to a desired testing state.
This exactly the situation we were encountering.
What's a good simple open source S3-style object store to use? Does one even exist? Do we need to start a project to (re-)write one in Rust?
- MinIO: more trouble than it's worth, as demonstrated here
- Radosgw of Ceph: not really suitable for a single server install
- Garage (https://garagehq.deuxfleurs.fr/): Their writing about the distributed design didn't convince me in 2022.
- https://lakefs.io/ Haven't studied
The situation is not as dire as it first appeared. While the messaging wasn't handled very well, where everything is settling out is good, with pluggable auth and an open source reference implementation: https://lakefs.io/blog/why-moving-acls-out-of-core-lakefs/
I would add https://github.com/gaul/s3proxy and https://lakefs.io/ to your list.
Also in the same vein, check out https://lakefs.io/