Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

111–120 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#111

Post by rrr_oh_man »

msoad wrote: post #31
rrr_oh_man wrote: post #30

Could anybody explain for the uninitiated?

It ensures that when you try to upload (or “put”) a new version of a file, the operation only succeeds if the file on the server still has the exact version (ETag) you specify. If someone else has updated the file in the meantime, your upload is blocked to prevent overwriting their changes. This is especially useful in scenarios where multiple users or processes are working on the same data, as it helps maintain cons…

Thank you! That was extremely helpful (and written in a way that is easy to understand)!

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#112

Post by tedk-42 »

vdm wrote: post #96
thayne wrote: post #84

Earlier quoted context omitted.

Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…

Don't the SDKs take care of computing the multi-part checksum during upload? > To create a trailing checksum when using an AWS SDK, populate the ChecksumAlgorithm parameter with your preferred algorithm. The SDK uses that algorithm to calculate the checksum for your object (or object parts) and automatically appends it to the end of your upload request. This behavior saves you time because Amazon S3 performs both the…

It does and has a good default. An issue I've come across though is you have the file locally and you want to check the e-tag value - you'll have to do this locally first and then compare the value to the S3 stored object.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#113

Post by zild3d »

torginus wrote: post #108
_zoltan_ wrote: post #107

Earlier quoted context omitted.

this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?

yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a doll…

Warm pools are a supported feature in AWS on auto scaling groups. Works as you're describing (have a pool of instances in stopped state ready to use, only pay for EBS volume if relevant) https://aws.amazon.com/blogs/compute/scaling-your-applicatio...

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#114

Post by merb »

torginus wrote: post #108
_zoltan_ wrote: post #107

Earlier quoted context omitted.

this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?

yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a doll…

I always thought that stopped instances will cost money as well?!

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#115

Post by belter »

torginus wrote: post #103

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one. This is one of those things…

If you use hourly billed machines...Sounds like the world most expensive semaphore :-)

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#116

Post by vdm »

vdm wrote: post #99
thayne wrote: post #84

Earlier quoted context omitted.

Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…

Ways to control etag/Additional Checksums without configuring clients: CopyObject writes a single part object and can read from a multipart object, as long as the parts total less than the 5 gibibyte limit for a single part. For future writes, s3:ObjectCreated:CompleteMultipartUpload event can trigger CopyObject, else defrag to policy size parts. Boto copy() with multipart_chunksize configured is the most convenient…

> Dividing object size by part size

Correction: and also part quantity (parsed from etag) for comparison

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#117

Post by viraptor »

amazingamazing wrote: post #80
viraptor wrote: post #79

Earlier quoted context omitted.

Someone made an informed technical bet that worked out. Sounds like HN material to me. (Also, is it really a useful ad if you can't easily use the product?)

Worked out how? There’s no implementation. It’s just conjecture.

It's right there:

> Our bet that S3 would get it in a reasonable time-frame worked out!

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#118

Post by vdm »

tedk-42 wrote: post #112
vdm wrote: post #96

Earlier quoted context omitted.

Don't the SDKs take care of computing the multi-part checksum during upload? > To create a trailing checksum when using an AWS SDK, populate the ChecksumAlgorithm parameter with your preferred algorithm. The SDK uses that algorithm to calculate the checksum for your object (or object parts) and automatically appends it to the end of your upload request. This behavior saves you time because Amazon S3 performs both the…

It does and has a good default. An issue I've come across though is you have the file locally and you want to check the e-tag value - you'll have to do this locally first and then compare the value to the S3 stored object.

https://github.com/peak/s3hash

It would be nice if this got updated for Additional Checksums.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#119

Post by papichulo2023 »

jayd16 wrote: post #62
CubsFan1060 wrote: post #25

I feel dumb for asking this, but can someone explain why this is such a big deal? I’m not quite sure I am grokking it yet.

When you upload a change you can know you're not clobbering changes you never saw.

I think is called write after write (WAW) if I remember correctly.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#120

Post by torginus »

merb wrote: post #114
torginus wrote: post #108

Earlier quoted context omitted.

yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a doll…

I always thought that stopped instances will cost money as well?!

You're only paying for the hard drive (and the VPC stuff, if you want to be pedantic). The downside is that if you try to start your instance, they might not start if AWS doesn't have the capacity (rare but have seen it happen, particularly with larger, more exotic instances.)
Post reply on HN