About synchronous disk replication • Hacker News Board

Re: About synchronous disk replication

#21

Post by nyrikki » Sat, Aug 31, 2024, 5:09 PM UTC

kobalsky wrote: post #16

karmakaze wrote: post #14

Earlier quoted context omitted.

I see your point, but calling this synchronous doesn't seem any farther off than a caching raid controller replying before writing.

From linked article: > If the disk replication status is catching up or degraded, then one of the zonal replicas is not updated with all the data. Any outage during this time in the zone of the healthy replica results in an unavailability of the disk until the healthy replica zone is restored. There isn't a binary log that the replicas can catch up to, if the healthy disk goes down you are out of luck. MySQL's semi-s…

MySQL semisynchronous is similar to a Cassandra consistency level of 1. That is typically a lens that helps.

As block devices aren't ACID, their challenges are greater.

But note the following from the MySQL docs, similar problems exist when you have to fail over with uncommitted transactions.

> With semisynchronous replication, if the source crashes and a failover to a replica is carried out, the failed source should not be reused as the replication source, and should be discarded. It could have transactions that were not acknowledged by any replica, which were therefore not committed before the failover.

Re: About synchronous disk replication

#22

Post by foota » Sat, Aug 31, 2024, 5:53 PM UTC

Balinares wrote: post #15

gregwebs wrote: post #3

This would be great for a lot of purposes but still seems to fall short for operating a db in which you don’t want any data loss. Although it is called synchronous replication, it appears to be asynchronous. That is the primary disk will acknowledge writes while the secondary may not have acknowledged them. To make something synchronous usually requires 2 secondaries where writes occur on 2/3 disks total: this allows…

Nope, this is in fact synchronous. It just degrades gracefully when one of the disaster domains is impacted. The main purpose here AIUI is availability, rather than durability. For durability, you'll indeed want something like 3-way replication. But that's a distinct problem. If durability is your concern but you're fine with the availability SLOs of a single disaster domain, then you don't need regional replication.

Is the idea that you'll check replication status after making each write? Or I guess you could do this at the application level, like after completing a transaction to e.g., sqlite check that there's at least one other zone caught up before acking whatever call the user made?

Re: About synchronous disk replication

#23

Post by leg » Sat, Aug 31, 2024, 6:56 PM UTC

jiggawatts wrote: post #6

Azure has had zone-redundant disks with 3-way synchronous writes since 2021: https://learn.microsoft.com/en-gb/azure/virtual-machines/dis... The same underlying system is also available for blob storage. It's strange that GCP and AWS are so far behind on this feature.

Regional Persistent Disk was in beta in 2019. Usability hiccups and other annoyances meant it only GA'd in 2023, but it's been used under CloudSQL for quite a while.

(I work on GCP storage)

Re: About synchronous disk replication

#24

Post by leg » Sat, Aug 31, 2024, 7:00 PM UTC

foota wrote: post #22

Balinares wrote: post #15

Earlier quoted context omitted.

Nope, this is in fact synchronous. It just degrades gracefully when one of the disaster domains is impacted. The main purpose here AIUI is availability, rather than durability. For durability, you'll indeed want something like 3-way replication. But that's a distinct problem. If durability is your concern but you're fine with the availability SLOs of a single disaster domain, then you don't need regional replication.

Is the idea that you'll check replication status after making each write? Or I guess you could do this at the application level, like after completing a transaction to e.g., sqlite check that there's at least one other zone caught up before acking whatever call the user made?

Many enterprise storage systems have the durability/availability tradeoff like these replicated disks when replicating outside of a single datacenter. (Oracle calls it "max availability": try to synchronously replicate, but if the remote side is offline, allow transactions to commit.) Real world banks run on these sorts of systems.

Users don't continuously check replication status. They rely on it being synchronous almost all the time.

3 way quorum replication is great, but you then need to send to more data centers, potentially affecting performance. There's a tradeoff.

(I work on GCP storage)

Re: About synchronous disk replication

#25

Post by gregwebs » Sat, Aug 31, 2024, 7:07 PM UTC

Balinares wrote: post #15

gregwebs wrote: post #3

This would be great for a lot of purposes but still seems to fall short for operating a db in which you don’t want any data loss. Although it is called synchronous replication, it appears to be asynchronous. That is the primary disk will acknowledge writes while the secondary may not have acknowledged them. To make something synchronous usually requires 2 secondaries where writes occur on 2/3 disks total: this allows…

Nope, this is in fact synchronous. It just degrades gracefully when one of the disaster domains is impacted. The main purpose here AIUI is availability, rather than durability. For durability, you'll indeed want something like 3-way replication. But that's a distinct problem. If durability is your concern but you're fine with the availability SLOs of a single disaster domain, then you don't need regional replication.

It’s synchronous much of the time but can become asynchronous at any moment? I would still call that asynchronous.

The availability story isn’t incredible either. Once the secondary is behind an outage of the primary means the system is no longer operational until that primary can be restored.

Three-way replication with raft and 2/3 or 3/5 acks is what modern distributed databases use. It’s for both availability and durability.

Re: About synchronous disk replication

#26

Post by time4tea » Sat, Aug 31, 2024, 8:59 PM UTC

Memories of EMC Symmetrix SRDF. Was a bit more expensive then though, but it was quite nice.

Re: About synchronous disk replication

#27

Post by jiggawatts » Sat, Aug 31, 2024, 10:37 PM UTC

leg wrote: post #23

jiggawatts wrote: post #6

Azure has had zone-redundant disks with 3-way synchronous writes since 2021: https://learn.microsoft.com/en-gb/azure/virtual-machines/dis... The same underlying system is also available for blob storage. It's strange that GCP and AWS are so far behind on this feature.

Regional Persistent Disk was in beta in 2019. Usability hiccups and other annoyances meant it only GA'd in 2023, but it's been used under CloudSQL for quite a while. (I work on GCP storage)

To be fair, it’s notable that Azure doesn’t publish any information about consistency guarantees (or lack thereof).

I did notice in an article about using blob witness for SQL clusters that they’re not all interchangeable.

Re: About synchronous disk replication

#28

Post by multani » Sun, Sep 01, 2024, 8:04 PM UTC

Is there any announcement for this new Hyper disk feature? That's something I was interested for a long time and I would have missed it if not for this HN post :D

As a follow-up of the availability topic: what's the best to simulate a zone outage on GCP?

Typicality, for this kind of disks (or previously "regional persistent disks"), you want to test that whatever automation you have for failing over to the secondary zone works correctly. It would be great also to be able to test other kind of failures at the zone level.