Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#61

Post by BikiniPrince »

texthompson wrote: post #47
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That's interesting. Would you want it to be something like a bucket setting, like "any time an object is uploaded, don't let an object write complete unless S3 verifies that a pre-defined hash function (like SHA256) is called to verify that the object's name matches the object's contents?"

You can already put with a sha256 hash. If it fails it just returns an error.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#63

Post by Salgat »

jiggawatts wrote: post #32
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

Isn't that the point of the metadata? Calculate the hash ahead of time and store it in the metadata as part of the atomic commit for the blob (at least for S3).

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#64

Post by flakes »

jiggawatts wrote: post #32
JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

That will probably never happen because of the fundamental nature of blob storage. Individual objects are split into multiple blocks, each of which can be stored independently on different underlying servers. Each can see its own block, but not any other block. Calculating a hash like SHA256 would require a sequential scan through all blocks. This could be done with a minimum of network traffic if instead of streamin…

You have just re-invented IPFS! https://en.m.wikipedia.org/wiki/InterPlanetary_File_System

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#65

Post by timmg »

CubsFan1060 wrote: post #28
Sirupsen wrote: post #27

Earlier quoted context omitted.

The short of it is that building a database on top of object storage has generally required a complicated, distributed system for consensus/metadata. CAS makes it possible to build these big data systems without any other dependencies. This is a win for simplicity and reliability.

Thanks! Do they mention when the comparison is done? Is it before, after, or during an upload? (For instance, if I have a 4tb file in a multi part upload, would I only know it would fail as soon as the whole file is uploaded?)

(I assume) it will fail if the eTag doesn't match -- the instance it got the header.

The main point of it is: I have an object that I want to mutate. I think I have the latest version in memory. So I update in memory and upload it to S3 with the eTag of the version I have and tell it to only commit if that is the latest version. If it "fails", I re-download the object, re-apply the mutation, and try again.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#66

Post by amazingamazing »

Sirupsen wrote: post #24

To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer

Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#67

Post by lxgr »

poincaredisk wrote: post #37
CubsFan1060 wrote: post #28

Earlier quoted context omitted.

Thanks! Do they mention when the comparison is done? Is it before, after, or during an upload? (For instance, if I have a 4tb file in a multi part upload, would I only know it would fail as soon as the whole file is uploaded?)

I imagine, for it to make sense, that the comparison is done at the last possible moment, before atomically swapping the file contents.

Practically, they could do both: Do an early reject of a given POST in case the ETag does not match, but re-validate this just before swapping out the objects (and committing to considering the given request as the successful one globally).

That said, I'm not sure if common HTTP libraries look at response headers before they're done posting a response body, or if that's even allowed/possible in HTTP? It seems feasible at a first glance with chunked encoding, at least.

Edit: Upon looking a bit, it seems that informational response codes, e.g. 100 (Continue) in combination with Expect 100-continue in the requests, could enable just that and avoid an extra GET with If-Match.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#68

Post by jiggawatts »

texthompson wrote: post #58
danielheath wrote: post #55

Earlier quoted context omitted.

S3 supports multipart uploads which don’t necessarily send all the parts to the same server.

Why does it matter where the bytes are stored at rest? Isn't everything you need for SHA-256 just the results of the SHA-256 algorithm on every 4096-byte block? I think you could just calculate that as the data is streamed in.

The data is not necessarily "streamed" in! That's a significant design feature to allow parallel uploads of a single object using many parts ("blocks"). See: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMu...

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#69

Post by User23 »

stevefan1999 wrote: post #54

So...are we closer to getting to use S3 as a...you guessed it...a database? With CAS, we are probably able to get a basic level of atomicity, and S3 itself is pretty durable, now we have to deal with consistency and isolation...although S3 branded itself as "eventually consistent"...

There was a great deal of interest in gossip protocols, eventual consistency, and such at Amazon in the mid oughts. So much so that they hired a certain Cornell professor along with the better part of his grad students to build out those technologies.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#70

Post by amazingamazing »

Ironically with this and lambda you could make a serverless sqlite by mapping pages to objects, using http range reads to read the db and lambda to translate queries to the writes in the appropriate pages via cas. Prior to this it would require a server to handle concurrent writers, making the whole thing a nonstarter for “serverless”.

Too bad performance would be terrible without a caching layer (ebs).

Post reply on HN