An open-source implementation of Amazon S3 - MinIO has had it for almost two years (relevant post: https://blog.min.io/leading-the-way-minios-conditional-write...). Strangely, Amazon is catching up just now.
Amazon S3 Adds Put-If-Match (Compare-and-Swap)
91–100 of 166 posts
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#92To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer
Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3
No one owes anyone open source. If they can make the business case work or if it works in their favor, sure.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#93I feel dumb for asking this, but can someone explain why this is such a big deal? I’m not quite sure I am grokking it yet.
When you upload a change you can know you're not clobbering changes you never saw.
Brilliant single line that is better than every other description above. Kudos.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#94An open-source implementation of Amazon S3 - MinIO has had it for almost two years (relevant post: https://blog.min.io/leading-the-way-minios-conditional-write... ). Strangely, Amazon is catching up just now.
That's not "strange" to me. Object storage has been a long time coming, and it's still being figured out: the entirely typical process of discovering useful and feasible primitives that expand applicability to more sophisticated problems. This is obviously going occur first in smaller and/or younger, more agile implementations, whereas AWS has the problem of implementing this at pretty much the largest conceivable scale with zero risk. The lag is, therefore, entirely unsurprising.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#95Earlier quoted context omitted.
Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3
https://github.com/slatedb/slatedb will, I expect, use this at some point. Object backed DB, which is open source.
Yes! I’m actively working on it, in fact. We’re waiting on the next release of the Rust `object_store` crate, which will bring support for S3’s native conditional puts.
If you want to follow along: https://github.com/slatedb/slatedb/issues/164
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#96Earlier quoted context omitted.
S3 has supported SHA-256 as a checksum algo since 2022. You can calculate the hash locally and then specify that hash in the PutObject call. S3 will calculate the hash and compare it with the hash in the PutObject call and reject the Put if they differ. The hash and algo are then stored in the object's metadata. You simply also use the SHA-256 hash as the key for the object. https://aws.amazon.com/blogs/aws/new-addit…
Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…
Don't the SDKs take care of computing the multi-part checksum during upload?
> To create a trailing checksum when using an AWS SDK, populate the ChecksumAlgorithm parameter with your preferred algorithm. The SDK uses that algorithm to calculate the checksum for your object (or object parts) and automatically appends it to the end of your upload request. This behavior saves you time because Amazon S3 performs both the verification and upload of your data in a single pass. https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#97It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.
While it can't be done server-side, this can be done straightforwardly in a signer service, and the signer doesn't need to interact with the payloads being uploaded. In other words, a tiny signer can act as a control plane for massive quantities of uploaded data.
The client sends the request headers (including the x-amz-content-sha256 header) to the signer, and the signer responds with a valid S3 PUT request (minus body). The client takes the signer's response, appends its chosen request payload, and uploads it to S3. With such a system, you can implement a signer in a lambda function, and the lambda function enforces the content-addressed invariant.
Unfortunately it doesn't work natively with multipart: while SigV4+S3 enables you to enforce the SHA256 of each individual part, you can't enforce the SHA256 of the entire object. If you really want, you can invent your own tree hashing format atop SHA256, and enforce content-addressability on that.
I have a blog post [1] that goes into more depth on signers in general.
[1] https://josnyder.com/blog/2024/patterns_in_s3_data_access.ht...
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#98Isn't this compare-and-set rather than compare-and-swap?
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#99Earlier quoted context omitted.
S3 has supported SHA-256 as a checksum algo since 2022. You can calculate the hash locally and then specify that hash in the PutObject call. S3 will calculate the hash and compare it with the hash in the PutObject call and reject the Put if they differ. The hash and algo are then stored in the object's metadata. You simply also use the SHA-256 hash as the key for the object. https://aws.amazon.com/blogs/aws/new-addit…
Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…
Ways to control etag/Additional Checksums without configuring clients:
CopyObject writes a single part object and can read from a multipart object, as long as the parts total less than the 5 gibibyte limit for a single part.
For future writes, s3:ObjectCreated:CompleteMultipartUpload event can trigger CopyObject, else defrag to policy size parts. Boto copy() with multipart_chunksize configured is the most convenient implementation, other SDKs lack an equivalent.
For past writes, existing multipart objects can be selected from inventory filtering ETag column length greater than 32 characters. Dividing object size by part size might hint if part size is policy.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#100Earlier quoted context omitted.
> Is there any reason you can't enforce that restriction on your side? I'd like to set IAM permissions for a role, so that that role can add objects to the content-addressible store, but only if their name matches the hash of their content. > Or are you saying you want S3 to automatically set the name for you based on the hash? I'm happy to name the files myself, if I can get S3 to enforce that. But sure, if it were…
I think you can presign PutObject calls that validate a particular SHA-256 checksum. An API endpoint, e.g. in a Lambda, can effectively enforce this rule. It unfortunately won’t work on multipart uploads except on individual parts.
Unfortunately, last I checked, the list of headers you're allowed to enforce for pre-signing does not include the hash.