Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

71–80 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#71

Post by UltraSane »

JoshTriplett wrote: post #21

It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.

S3 has supported SHA-256 as a checksum algo since 2022. You can calculate the hash locally and then specify that hash in the PutObject call. S3 will calculate the hash and compare it with the hash in the PutObject call and reject the Put if they differ. The hash and algo are then stored in the object's metadata. You simply also use the SHA-256 hash as the key for the object.

https://aws.amazon.com/blogs/aws/new-additional-checksum-alg...

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#72

Post by UltraSane »

mdavidn wrote: post #52
JoshTriplett wrote: post #39

Earlier quoted context omitted.

> Is there any reason you can't enforce that restriction on your side? I'd like to set IAM permissions for a role, so that that role can add objects to the content-addressible store, but only if their name matches the hash of their content. > Or are you saying you want S3 to automatically set the name for you based on the hash? I'm happy to name the files myself, if I can get S3 to enforce that. But sure, if it were…

I think you can presign PutObject calls that validate a particular SHA-256 checksum. An API endpoint, e.g. in a Lambda, can effectively enforce this rule. It unfortunately won’t work on multipart uploads except on individual parts.

The hash of multipart uploads is simply the hash of all the part hashes. I've been able to replicate it.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#73

Post by captn3m0 »

amazingamazing wrote: post #70

Ironically with this and lambda you could make a serverless sqlite by mapping pages to objects, using http range reads to read the db and lambda to translate queries to the writes in the appropriate pages via cas. Prior to this it would require a server to handle concurrent writers, making the whole thing a nonstarter for “serverless”. Too bad performance would be terrible without a caching layer (ebs).

For read heavy workloads, you could cache the results at cloudfront. Maybe we will someday see Wordpress-on-Lambda-to-Sqlite-over-S3.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#74

Post by tonymet »

akira2501 wrote: post #17
tonymet wrote: post #14

good example of how a simple feature on the surface (a header comparison) requires tremendous complexity and capacity on the backend.

S3 is rated as "durable" as opposed to "best effort." It has lots of interesting guarantees as a result.

Also they are faithful to their consistency commitments

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#76

Post by UltraSane »

offmycloud wrote: post #13

If the default ETag algorithm for non-encrypted, non-multipart uploads in AWS is a plain MD5 hash, is this subject to failure for object data with MD5 collisions? I'm thinking of a situation in which an application assumes that different (possibly adversarial) user-provided data will always generate a different ETag.

The default Etag is used to detect bit errors and and MD5 is fine for that. S3 does support using SHA256 instead.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#78

Post by UltraSane »

fragmede wrote: post #40
mannyv wrote: post #38

Earlier quoted context omitted.

GCP still doesn't have triggers out of beta last time i checked (which was a while ago).

Gmail was in beta for five years, I don't think that label really means anything.

It means that Google doesn't want to offer an SLA

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#79

Post by viraptor »

amazingamazing wrote: post #66
Sirupsen wrote: post #24

To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer

Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3

Someone made an informed technical bet that worked out. Sounds like HN material to me. (Also, is it really a useful ad if you can't easily use the product?)

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#80

Post by amazingamazing »

viraptor wrote: post #79
amazingamazing wrote: post #66

Earlier quoted context omitted.

Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3

Someone made an informed technical bet that worked out. Sounds like HN material to me. (Also, is it really a useful ad if you can't easily use the product?)

Worked out how? There’s no implementation. It’s just conjecture.
Post reply on HN