Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#41

Post by anotheraccount9 »

JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

Could you use a meta field from the object and save the hash in it, running a compare from it?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#42

Post by [deleted] »

johnrob wrote: post #16
paulddraper wrote: post #12

Earlier quoted context omitted.

So....given CAP, which one did they give up

I’d wager that the algorithm is slightly eager to throw a consistency error if it’s unable to verify across partitions. Since the caller is naturally ready for this error, it’s likely not a problem. So in short it’s the P :)

[deleted]

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#43

Post by rhaen »

johnrob wrote: post #23
alanyilunli wrote: post #18

Earlier quoted context omitted.

Shouldn't that be the A then? Since the network partition is still there but availability is non-guaranteed.

Yes, definitely. Good point (I was knee jerk assuming the A is always chosen and the real “choice” is between C and P).

Well, P isn't really much of a choice, I don't think you can opt out of acts of god.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#45

Post by the_arun »

paulddraper wrote: post #12
1a527dd5 wrote: post #8

Be still my beating heart. I have lived to see this day. Genuinely, we've wanted this for ages and we got half way there with strong consistency.

So....given CAP, which one did they give up

I thought they have implemented Optimistic locking now to coordinate concurrent writes. How does it change anything in CAP?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#47

Post by texthompson »

JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That's interesting. Would you want it to be something like a bucket setting, like "any time an object is uploaded, don't let an object write complete unless S3 verifies that a pre-defined hash function (like SHA256) is called to verify that the object's name matches the object's contents?"

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#48

Post by texthompson »

jiggawatts wrote: post #32
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Why would you PUT an object, then download it again to a central server in the first place? If a service is accepting an upload of the bytes, it is already doing a pass over all the bytes anyway. It doesn't seem like a ton of overhead to calculate SHA256 in the 4092-byte chunks as the upload progresses. I suspect that sort of calculation would happen anyways.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#49

Post by fwip »

rhaen wrote: post #43
johnrob wrote: post #23

Earlier quoted context omitted.

Yes, definitely. Good point (I was knee jerk assuming the A is always chosen and the real “choice” is between C and P).

Well, P isn't really much of a choice, I don't think you can opt out of acts of god.

You can design to minimize P, though. For instance, if you have all the services running on the same physical box, and make people enter the room to use it instead of over the Internet, "partition" becomes much less likely. (This example is a bit silly.)

But you're right, if you take a broad view of P, the choice is really between consistency and availability.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#50

Post by btown »

johnrob wrote: post #23
alanyilunli wrote: post #18

Earlier quoted context omitted.

Shouldn't that be the A then? Since the network partition is still there but availability is non-guaranteed.

Yes, definitely. Good point (I was knee jerk assuming the A is always chosen and the real “choice” is between C and P).

https://tqdev.com/2024-the-p-in-cap-is-for-performance is a really interesting take on this as a response to https://blog.dtornow.com/the-cap-theorem.-the-bad-the-bad-th... - essentially, the only way to get CA is if you're willing to say that every request will succeed eventually, but it might take an unbounded amount of time for partitions to heal, and you have to be willing to wait indefinitely for that to happen. Which can indeed make sense for asynchronous messaging, but not for real-time applications as we think about them in the modern day. In practice, if you're talking about CAP for high-performance systems, you're choosing either CP or AP.
Post reply on HN