This article is a bit hard for me to grasp the main ideas of because, given Cloudflare's requirements (e.g. no strong leaders), it immediately seems like they should be comparing to leaderless protocols like Paxos-class algorithms. Comparing to Raft and saying it's better because Meerkat is leaderless is confusing, because Raft is an adjustment to Paxos to specifically have strong leaders. So I'm 3/4 the way into the article and I don't see what's unique here.
I think the unique idea here is supposed to be QuePaxa's idea of avoiding timeouts for ensuring liveness. The actual discussion of QuePaxa is limited to one paragraph at the end, and tbh only a couple sentences of that paragraph.
I feel like the article could've been titled "Consensus protocols and linearizability: a brief explainer", or "Paxos vs Raft", or similar. It just doesn't feel like it communicates what it claims to communicate, and is a bit confused on who its audience is, just IMO.
Agreed on - it took a long time to get to the “so what is new here” vs the broader topic of distributed consensus. Stylistically would prefer more upfront “here is what is novel here”
Worth noting if it wasn't obvious from the article that Cloudflare did not develop QuePaxa. It's from an SOSP paper back in 2023[0]. The article is discussing what is the first known large-scale public deployment of the protocol.
Right. But given that the entire point revolves around QuePaxa, it's strange to see no discussion on it. If that weren't the point, the article would be "Why Cloudflare implemented and deployed Paxos".
Which would also be a good read, but this article also isn't that. It doesn't discuss their experience deploying the protocol, aside from the following statement:
> Meerkat is not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate.
I think it would've been more interesting to read why Cloudflare chose the specific algorithm they did, see an example of a pathological but common situation Cloudflare sees at their scale that makes other protocols unsuitable for them, therefore they made X choice and this led to Y gains in production (or on dummy workloads, or whatever). As it stands, there's nothing here actually specific to Cloudflare's workload or deployment. It doesn't even state their use-case beyond "small pieces of control plane state (e.g., leadership for replicated databases)"
if youve ever fought a raft cluster on a bad network with leaders flapping, elections storming and latency spiking this genuinely doesnt seem that bad. i believe this will be very useful to those dealing with messy networks
TBH I’m doubtful of most people building their own crypto libs and distributed consensus implementations. But maybe cloudflare can pull it off. Good to see them pushing the state of distributed consensus.
Take aways:
* it’s not in prod yet. I suspect those many round trips are going to get expensive on median aka typical redistributed deployments. Curious to see how it goes once in the wild.
* they say it isn’t likely suitable for eg databases.
* they talk about formal verification, which is good and feels appropriate.
Consensus is one of those areas where the interesting engineering and the number of
people who actually need it are inversely correlated. Most "we need distributed
coordination" turns out to be "we need one writer and a lock," which a single
Postgres hands you for free: advisory locks, SELECT ... FOR UPDATE, SKIP LOCKED for
work distribution. Linearizability without running Raft..
The real threshold is multi-region writes on a hard latency budget, and even then a
single-region primary plus accepting cross-region read latency beats eating a
consensus round-trip on every write for a lot of teams. Curious what workload pushed
you past single-primary - usually a better story than the impl itself.
This article is a bit hard for me to grasp the main ideas of because, given Cloudflare's requirements (e.g. no strong leaders), it immediately seems like they should be comparing to leaderless protocols like Paxos-class algorithms. Comparing to Raft and saying it's better because Meerkat is leaderless is confusing, because Raft is an adjustment to Paxos to specifically have strong leaders. So I'm 3/4 the way into the article and I don't see what's unique here.
I think the unique idea here is supposed to be QuePaxa's idea of avoiding timeouts for ensuring liveness. The actual discussion of QuePaxa is limited to one paragraph at the end, and tbh only a couple sentences of that paragraph.
I feel like the article could've been titled "Consensus protocols and linearizability: a brief explainer", or "Paxos vs Raft", or similar. It just doesn't feel like it communicates what it claims to communicate, and is a bit confused on who its audience is, just IMO.
Agreed on - it took a long time to get to the “so what is new here” vs the broader topic of distributed consensus. Stylistically would prefer more upfront “here is what is novel here”
Worth noting if it wasn't obvious from the article that Cloudflare did not develop QuePaxa. It's from an SOSP paper back in 2023[0]. The article is discussing what is the first known large-scale public deployment of the protocol.
[0] https://dl.acm.org/doi/10.1145/3600006.3613150
Right. But given that the entire point revolves around QuePaxa, it's strange to see no discussion on it. If that weren't the point, the article would be "Why Cloudflare implemented and deployed Paxos".
Which would also be a good read, but this article also isn't that. It doesn't discuss their experience deploying the protocol, aside from the following statement:
> Meerkat is not deployed to production, but we have run multiple proofs-of-concept with up to 50 replicas distributed around the world, to great success. Leaders in our proof-of-concept clusters constantly fail, and the cluster keeps operating with no increase in error-rate.
I think it would've been more interesting to read why Cloudflare chose the specific algorithm they did, see an example of a pathological but common situation Cloudflare sees at their scale that makes other protocols unsuitable for them, therefore they made X choice and this led to Y gains in production (or on dummy workloads, or whatever). As it stands, there's nothing here actually specific to Cloudflare's workload or deployment. It doesn't even state their use-case beyond "small pieces of control plane state (e.g., leadership for replicated databases)"
Illustrated with meercats looking in different directions.
if youve ever fought a raft cluster on a bad network with leaders flapping, elections storming and latency spiking this genuinely doesnt seem that bad. i believe this will be very useful to those dealing with messy networks
TBH I’m doubtful of most people building their own crypto libs and distributed consensus implementations. But maybe cloudflare can pull it off. Good to see them pushing the state of distributed consensus.
Take aways:
* it’s not in prod yet. I suspect those many round trips are going to get expensive on median aka typical redistributed deployments. Curious to see how it goes once in the wild.
* they say it isn’t likely suitable for eg databases.
* they talk about formal verification, which is good and feels appropriate.
Looking forward to seeing more!
> I’m doubtful of most people building their own crypto libs and distributed consensus implementations. But maybe cloudflare can pull it off.
Who else than Cloudflare (or similar company in expertise and size) would be a better fit to implement distributed consensus?
Cloudflare did not develop QuePaxa, although I'm sure they did a lot of work to operationalize it. The original protocol is from a SOSP 2023 paper.
https://dl.acm.org/doi/10.1145/3600006.3613150
Consensus is one of those areas where the interesting engineering and the number of people who actually need it are inversely correlated. Most "we need distributed coordination" turns out to be "we need one writer and a lock," which a single Postgres hands you for free: advisory locks, SELECT ... FOR UPDATE, SKIP LOCKED for work distribution. Linearizability without running Raft..
The real threshold is multi-region writes on a hard latency budget, and even then a single-region primary plus accepting cross-region read latency beats eating a consensus round-trip on every write for a lot of teams. Curious what workload pushed you past single-primary - usually a better story than the impl itself.