Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

231–240 of 240 posts

Re: S3 Strong Consistency

#231

Post by pricechild »

johncolanduoni wrote: post #200
pricechild wrote: post #195

Earlier quoted context omitted.

"almost certainly not fully partition tolerant at the node level" Whoa hang on! You can't just say you're not tolerant of partitions... I'm struggling to find the best post on aphyr.com about this but https://aphyr.com/posts/325-comments-on-you-do-it-to is a good one, specifically the line: "CP and AP are upper bounds: systems can provide C or A during a partition, but might provide neither." In short, Partitions hap…

I’m not claiming it’s a CA system, and the terminology “partition intolerant” is not verboten by Kyle Kingsbury. From your link: “Specifically, partition-intolerant systems must sacrifice invariants when partitions occur. Which invariants?” The answer in this case is that availability is sacrificed, unless Amazon is making a very misleading claim of strong consistency (per the submission title/link). So it’s CP. In c…

Cool, sorry to put words in your mouth.

I guess the simple answer to the GP is "Availability is likely sacrificed", and the clue's in the submission title being all about consistency. I haven't looked too deep into it and am sure there are also still Consistency sacrifices too.

I'm still a little niggled though... "almost certainly not fully partition tolerant at the node level and requires some sort of quorum" - what does this actually mean? I'm confused why you raise the tolerance of a single node to partitions, when the partitions by definition require >1 node?

Re: S3 Strong Consistency

#232

Post by valenterry »

benlivengood wrote: post #223
valenterry wrote: post #203

Earlier quoted context omitted.

Then what happens in case of network partitions?

One side of the partition has quorum and the other does not. Processing keeps happening on the partition with quorum. A practical example I worked on is Google's Photon; multi-region logjoining with Paxos for consistency. When the network caused a region to become unavailable all the logjoining work happened in the other regions. Resources were provisioned for enough excess capacity to run at full speed in the case o…

You are describing the case where there are 2 groups that are isolated/partitioned.

But what if there are more than 2?

Re: S3 Strong Consistency

#233

Post by YZF »

throw93 wrote: post #218
YZF wrote: post #90

Earlier quoted context omitted.

While I don't know the specifics I can say that availability is down to engineering practices. Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency. The availability aspect of this is just engineering, making sure you're never down to less than a qu…

>> Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency. If reads are reading only from a quorum of nodes how do you guarantee they have latest data? In theory, while a node is servicing a read request wouldn’t you need to query “all” other nodes to…

You write to a quorum of nodes and you read from a quorum of nodes. Latest is determined by a clock. Let's say you wrote to 2 out of 3 nodes, and read from 2 out of 3 nodes, one of those 2 you read from is going to have the latest data. This is how consistency works in Cassandra e.g.

There are other models, for example by using a consensus protocol.

Re: S3 Strong Consistency

#234

Post by YZF »

valenterry wrote: post #123
YZF wrote: post #90

Earlier quoted context omitted.

While I don't know the specifics I can say that availability is down to engineering practices. Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency. The availability aspect of this is just engineering, making sure you're never down to less than a qu…

> (of their metadata database) With that you have just moved the question to: how do they ensure that the metadata database is available _and_ strongly consistent at the same time for all the requests? Because the metadata database is certainly also a distributed one, hence you need to query _all_ the nodes or can end up in a split-brain situation and lose consistency (or availability if you choose to down the system…

All you really care about is that P(split_brain) An example I like to use is integrated circuits, like your CPU, sure, it can fail in a mode where you lose the internal connectivity between the transistors, and then your CPU is dead. It's a very complex distributed system where engineers have basically taken out any practical possibility of an internal failure. So just because CAP says that if you lose connectivity to some portion of transistors you can't maintain consistency or availability (which is true) doesn't mean a thing in practice. Nobody even worries about how to keep a CPU going if you lose one core.

Re: S3 Strong Consistency

#235

Post by stubish »

sleepydog wrote: post #142
stubish wrote: post #85

Once libraries and tools start relying on this, it is going to make life interesting to the S3-compatible players. The API remains the same, but the behavior is quite different.

Google Cloud storage has behaved this way for years. According to other comments, so does Azure.

It is the smaller players I'm thinking of, which needed S3 compatibility for uptake. Openstack Swift, Backblaze, Ceph... I am not sure if any of these suddenly stopped being S3 compatible due to this announcement.

Re: S3 Strong Consistency

#236

Post by scarface74 »

OJFord wrote: post #24
mholt wrote: post #20

Would this enable atomic operations with S3 (e.g. "put object only if it does not exist")?

Could you not already do that by specifying ETag?

The Etag is not a reliable hash of the file contents. It will be different if the file was uploaded in multiple parts like using the CLI than if it was moved as one operation like copying from one bucket to another. You can tell the difference because the Etag will have a “-“ in it if it was uploaded as a multipart upload.

Re: S3 Strong Consistency

#237

Post by valenterry »

YZF wrote: post #234
valenterry wrote: post #123

Earlier quoted context omitted.

> (of their metadata database) With that you have just moved the question to: how do they ensure that the metadata database is available _and_ strongly consistent at the same time for all the requests? Because the metadata database is certainly also a distributed one, hence you need to query _all_ the nodes or can end up in a split-brain situation and lose consistency (or availability if you choose to down the system…

All you really care about is that P(split_brain) An example I like to use is integrated circuits, like your CPU, sure, it can fail in a mode where you lose the internal connectivity between the transistors, and then your CPU is dead. It's a very complex distributed system where engineers have basically taken out any practical possibility of an internal failure. So just because CAP says that if you lose connectivity t…

> All you really care about is that P(split_brain) In other words: availability suffers from it.

> An example I like to use is integrated circuits, like your CPU, sure, it can fail in a mode where you lose the internal connectivity between the transistors, and then your CPU is dead

In that case the system is dead, it doesn't become inconsistent. I get your point though, you can reduce the chance-of-happening for these cases, but this orthogonal for the CAP. The CAP says you have to balance between availability and consistency (assuming partitioning is required here). You cannot have both at 100%, no matter how efficient you get.

And if the efficiency stays the same (which we should assume when comparing S3 with and without strong consistency) then giving better consistency guarantees inevitably must reduce availability - that's what my point is.

Re: S3 Strong Consistency

#238

Post by neverartful »

stubish wrote: post #235
sleepydog wrote: post #142

Earlier quoted context omitted.

Google Cloud storage has behaved this way for years. According to other comments, so does Azure.

It is the smaller players I'm thinking of, which needed S3 compatibility for uptake. Openstack Swift, Backblaze, Ceph... I am not sure if any of these suddenly stopped being S3 compatible due to this announcement.

Ceph has strong consistency by design. S3 compatibility in Ceph comes via the Rados Gateway (RGW) which just sits atop normal Ceph. This means that the strong consistency is carried over to RGW. So, no change for Ceph.

OpenStack Swift has an eventually consistent design. If the Swift community desires to maintain strict compatibility with S3 regarding consistency model, it would be a challenge. Despite being a challenge, I would never bet against the Swift developers. If they want to do it, I'm quite sure that they can. Having said all of that, I doubt that the Swift community would consider this a burning issue that needs to be solved. They've existed for over 10 years with an eventually consistent model and their user base is well aware of it.

Re: S3 Strong Consistency

#239

Post by johncolanduoni »

pricechild wrote: post #231
johncolanduoni wrote: post #200

Earlier quoted context omitted.

I’m not claiming it’s a CA system, and the terminology “partition intolerant” is not verboten by Kyle Kingsbury. From your link: “Specifically, partition-intolerant systems must sacrifice invariants when partitions occur. Which invariants?” The answer in this case is that availability is sacrificed, unless Amazon is making a very misleading claim of strong consistency (per the submission title/link). So it’s CP. In c…

Cool, sorry to put words in your mouth. I guess the simple answer to the GP is "Availability is likely sacrificed", and the clue's in the submission title being all about consistency. I haven't looked too deep into it and am sure there are also still Consistency sacrifices too. I'm still a little niggled though... "almost certainly not fully partition tolerant at the node level and requires some sort of quorum" - wha…

When I say “at the node level” I’m trying to draw a distinction with the service level view where a client is talking to S3 as a (from their perspective) single entity per region. So a better way of putting it may be the internal view of the service, where partitions are between nodes that are part of S3 itself (as opposed to between a client and S3) and they affect the overall health of the system. I’m not referring to a single node being partition resistant (hence why I said node level, and never mentioned single) which as you said wouldn’t make any sense.

Re: S3 Strong Consistency

#240

Post by nhumrich »

oblio wrote: post #217
nhumrich wrote: post #79

Earlier quoted context omitted.

At the benefit of added availability? absolutely. This is the entire premise of mongodb.

Availability? I thought Mongo was fast and scalable, I don't remember anyone recommending it for higher availability.

> Fast and scalable

> Availability

These are synonyms. How do you scale? Horizontally. How you do it without degrading performance, is the same way you increase availability. Eventual consistency.

Post reply on HN