Murat Demirbas

There’s also the part where the disaster recovery site apparently fell over under the load (which, OK, is a thing that might happen) and they needed to code up limits on the fly (and that is not OK; I don’t have the slightest idea how one might test this, but if you’re building a “disaster” site it seems like you’d need to figure it out):

> When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. We implemented rate limits to get the request volume under control.

This seems not to be mentioned in the bullet points at the end of the text (which are otherwise reasonable).

And now I’m curious—how do you design cold failover when the system is complex enough to be metastable[1] and you can’t afford to test it on live traffic? I can guess which techniques you could use to build it, it’s the design and testing part (knowing the techniques actually work in your situation) that’s the problem.

One other thing that seems to have gone completely unmentioned:

> Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. [... W]e made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe.

Why did the decision take so long? I can imagine it can’t be made lightly, but two hours seems like too much hesitation, even if there was an expectation that power would be restored imminently for most of…

“Paxos is one of the first and most celebrated ideas that laid the foundation for how different things come to an agreement asynchronously”

“Most—if not all—consensus algorithms fundamentally derive concepts from Paxos”

It's not well known (even in the field of distributed systems) that in fact Viewstamped Replication pioneered the field of consensus in 1988, a year before Paxos, with an intuitive algorithm essentially identical to Raft, so that Raft is more within the family and tradition of Viewstamped Replication (as is Multi-Paxos state machine replication) than the other way round.

Viewstamped Replication is significantly easier to implement correctly in the presence of real world storage faults, which formal proofs often do not take into account in their fault model. For example, Paxos and Raft proofs typically assume only pristine stable storage, and the correctness unfortunately rapidly breaks down when physical disks are used, since these may misdirect reads and writes.

UW-Madison have done some excellent research in this area in particular, the intersection of consensus protocol correctness and storage faults, and I would recommend reading "Protocol-Aware Recovery for Consensus-Based Storage" if you'd like to dive into some of this: https://www.usenix.org/conference/fast18/presentation/alagappan

Aleksey Charapko's DistSys Reading Group also recently covered Viewstamped Replication as a foundational paper, focusing on the crystal clear 2012 revision by Barbara…

I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. As system designers, we're very used to thinking about systems as 'stable' and 'unstable', where stability is good, and instability is bad. What this paper points out is that many kinds of distributed systems have multiple 'stable' modes, some of which are modes where the system is stable (in a control theory sense), but not doing any useful work from the client's perspective. This is dangerous, because the system won't kick itself out of this "stable but down" mode without something changing: human input, a control plane taking action, etc.

I don't think this paper covers anything particularly new, but writing it down in this form, with the evidence they present, is very valuable. Hopefully this paper will deepen the conversation about applying control theory to distributed systems design and control problems, and allow a more theoretical approach to be taken to the design of these systems to avoid common causes of instability and bistability.

One of the authors has a great summary of the paper on his blog: http://charap.co/metastable-failures-in-distributed-systems/

I wrote a summary and discussion too: https://brooker.co.za/blog/2021/05/24/metastable.html

While reading Aleksey’s blog[0] linked from yours it struck me that this sounds a great deal like (real world) traffic engineering. A backup occurs, the traffic grinds to a halt, and after the blockage is removed it takes a very long time for the system to recover. If traffic is heavy enough, even just a small slowdown is enough to cause prolonged pain.

[0]: http://charap.co/metastable-failures-in-distributed-systems/

(Having said that, my traffic engineering cluefulness is roughly at the “casual reader glancing at headlines” level, if that, but I found the comparison interesting.)

I’m unfamiliar with HotOS, so I’m curious about the context behind this sentence:

> But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.

> I don't care about consistency.

Fair nuff, I asked because I was curious if you were assuming that eventual consistency was necessary for high availability.

> Write rate isn't really much of a consideration since it's almost embarrassingly low (hundreds per second).

Yup, looks like my concern above is the case. A single consensus replica set will do that no problem. I'd use raft if you're in a single geo location or epaxos otherwise (the epaxos source is a little early and needs eyeballs/production diversity testing, but the algorithm is absolutely solid). There's no reason to borrow all the troubles of CRDT's when it's not needed.

Edit:

Example from the epaxos paper, this is latency vs throughput for 3 replicas in one geo location: http://charap.co/wp-content/uploads/2015/10/latency_over_throughput.png