Authzed Blog

> Additionally, there are folks using SpiceDB today by replicating denormalized checks back into their database (e.g. Postgres) or search index (e.g. Elastic) so that you can filter them natively. This is the combination of the aforementioned Lookup APIs with our Watch API. While this strategy requires moving parts, it is necessary beyond a particular scale which is well beyond the point at which policy engines typically fall over.

Would you say that because of this, Zanzibar engines like spicedb only become useful on systems of a certain size / complexity? Fundamentally you run into data synchronization issues whether you are syncing denormalized data back to your db via Watch or whether you write the relationships to both data stores in the first place. This article[0] on the latter topic touches on this, but brushes over some tricker parts of implementing such a thing correctly (eg. 2 writes section only covers insert not update or delete which is generally less harmful to have a ghost update that persists in spicedb, streaming updates brushes over some major footguns).

Granted there's nothing unique to spicedb in this sort of complexity, but by nature of being a db, using spicedb mandates that users must take on the complexity.

Is it then fair to say that it is appropriate to use spicedb once a project reaches a certain size / complexity, or would you expect a startup to adopt it from the beginning?

[0] https://authzed.com/blog/writing-relationships-to-spicedb

Reconciling externalized authz with search is actually quite a challenging problem. For standard externalized authz, the recommendation is some form of pre-filtering or post-filtering [1], for which we actually built LookupResources (pre-filtering) and CheckBulkPermission (post-filtering) into SpiceDB.

However, as you mentioned, life is easier when the main database can handle everything, so we actually built a solution in that space called Materialize [2], which heavily denormalizes the authorization data and allows for joining within application databases such as Postgres. My colleague Evan actually put together a really cool video about using it with Gitea [3].

Recognizing that even with Materialize, however, the need to consume events can be a bit annoying, I've been doing some work to allow Postgres itself to do native JOINs against SpiceDB (and other operations). I demo it briefly in our recent announcements video [4] and I think it effectively solves this problem within Postgres, while still allowing for all the benefits (scale, performance, redundancy, distribution) of externalized authz.

[1]: https://authzed.com/docs/spicedb/modeling/protecting-a-list-endpoint

[2]: https://authzed.com/products/authzed-materialize

[3]: https://www.youtube.com/live/u3i1SEd9Ll8?si=mCz5mZterxthoEwj

[4]: https://www.youtube.com/live/uz_gxz3whS0?si=g4NUZAIltYVyFzYj&t=3499

Disclaimer: I'm cofounder and CTO at AuthZed and we build SpiceDB and Materialize

Authzed (YC W21) | Senior Site-Reliability Engineer (SRE) | NYC, Remote USA | Full-Time | https://authzed.com/

20 milliseconds at the 99.5 percentile and five-nines (99.999) of uptime. Those are our goals for our globally-distributed permissions system as a service. If you’re interested in helping us achieve these goals, we would like to talk. We’re currently using best-in-class open source solutions on the public cloud for delivering this vision: Kubernetes, CockroachDB, PostgreSQL, Prometheus, Pulumi, Cuelang, Let’s Encrypt, etc.

As part of our small team, you will help to build our application platform, delivery pipelines, data architecture, and the application itself. You should be passionate about both software development and infrastructure. In this role you will be required to code. You will also participate in our on-call rotation which is responsible for the health of our highly-available globally-distributed service.

About the Company

At Authzed, we’re solving application permissions. We’re building a highly available, highly reliable, multi-tenant permissions system as a service which is deployed globally. Modeled after Google’s Zanzibar, our core open-source component SpiceDB is revolutionizing the way developers build powerful, flexible permissions into any application. With our founding team formerly of Red Hat and CoreOS, open-source is deeply ingrained in the fabric of our company.

https://www.workatastartup.com/jobs/47704

Full disclosure: I'm a founder of Authzed (YC W21)[0], which is a productized Zanzibar system.

> just want to understand what makes it "highly scalable", as it isn't clear from the article.

While this post might not cover in depth why their system is scalable, the Zanzibar paper[1] and this post[2] about Zanzibar covers why similar systems are scalable. A oversimplification would be to say that finding a path between two nodes in a graph can be recursively broken down and performed in parallel -- the Authzed implementation looks roughly similar to a map reduce.

>Are ALL permissions stored as an Object::relation pair? i.e. do you need to register permissions for all new entities relationships, or do you have some way of storing more dynamic permissions?

Zanzibar implementations do have graph schemas (called Namespace Configs in the paper) which offer set operations for computing dynamic relationships. When you create a new user or object in your app, you typically also write a few relationships into Zanzibar and the rest of their permissions are implicitly granted through the computed definitions in your schema.

If you're interested in learning more about Zanzibar, we're giving a PapersWeLove talk that will be streamed on Twitch next week: https://www.meetup.com/papers-we-love/events/278148236/

[0]: https://authzed.com

[1]: https://research.google/pubs/pub48190/

[2]: https://authzed.com/blog/what-is-zanzibar/

This might just be a language issue, because different sources use different words for consistency guarantees.

> Accord only avoids enforcing an ordering (2) on transactions that are commutative

If the committed state in the database records the transactions in a different order than an external observer could have observed them being accepted by the database, then you don't have what Spanner calls "external consistency" - I thought this is what you mean when you say "global strict serializability", but now I'm not so sure.

Cockroach does enforce this property, but only for transactions that touch the same (key-value layer) keys. They talk about this a bit in their article on "life without atomic clocks"[0].

In SpiceDB[1], we take advantage of this property (when Cockroach is selected as the backing datastore) to give us external consistency by forcing all transactions to overlap. This prevents the New Enemy Problem[2], which is what Google calls the problem of transaction ordering when seen from an authorization perspective.

Your description of Accord sounded very similar to how Cockroach works from this perspective, but there may be some subtlety that I am missing. Cockroach exposes snapshots via "as of system time" queries, which can expose "bad" ordering back to the client in future queries - if Accord doesn't allow snapshot queries then I suppose it wouldn't be an issue.

[0]:…

Agreed! But you're glossing over the Zanzibar point of view on this topic, which falls back to dual-writes. That approach has a lot of downsides: "Unfortunately, when making writes to multiple systems, there are no easy answers."[0]

Having spoken with the actual creators of Zanzibar, they lament the massive challenge this design presents and the heroics they undertook over 7+ years at Google to overcome them.

By contrast, we're seeing lots of the best tech companies opt for approaches that let them leave the data in their source database wherever and as much as possible [1]

[0] https://authzed.com/blog/the-dual-write-problem

[1] https://www.osohq.com/post/local-authorization

I'm founder of Oso btw.