Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 Adds Put-If-Match (Compare-and-Swap)

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 166 posts

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#101

Post by CobrastanJorji »

Sirupsen wrote: post #24

To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer

I'm glad that bet worked out for you, but what made you think one year ago that S3 would introduce it soon that was untrue for the previous 15 years?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#102

Post by CobrastanJorji »

CubsFan1060 wrote: post #25

I feel dumb for asking this, but can someone explain why this is such a big deal? I’m not quite sure I am grokking it yet.

It is often very important to know, when you write an object, what the previous state was. Say you sold plushies and you had 100 plushies in a warehouse. You create a file "remainingPlushies.txt" that stores "100". If somebody buys a plushie, you read the file, and if it's bigger than 0, you subtract 1, write the new version of the file, and okay the sale.

Without conditional writes, two instances of your application might both read "100", both subtract 1, and both write "99". If they checked the file afterward, both would think everything was fine. But things aren't find because you've actually sold two.

The other cloud storage providers have had these sorts of conditional write features since basically forever, and it's always been really weird that S3 has lacked them.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#103

Post by torginus »

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives.

My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one.

This is one of those things that I never advertised out of professional shame, but it works, its bulletproof and dead simple and does not require additional infra to work.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#104

Post by CobrastanJorji »

offmycloud wrote: post #13

If the default ETag algorithm for non-encrypted, non-multipart uploads in AWS is a plain MD5 hash, is this subject to failure for object data with MD5 collisions? I'm thinking of a situation in which an application assumes that different (possibly adversarial) user-provided data will always generate a different ETag.

With Google Cloud Storage, you can solve this by conditionally writing based on the "generation number" of the object, which always increases with each new write, so you can know whether the object has been overwritten regardless of its contents. I think Azure also has an equivalent.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#105

Post by BrandonY »

mannyv wrote: post #38
sillysaurusx wrote: post #7

Finally. GCP has had this for a long time. Years ago I was surprised S3 didn’t.

GCP still doesn't have triggers out of beta last time i checked (which was a while ago).

We do have Cloud Run Functions that trigger on Cloud Storage events, as well as Cloud Pub/Sub notifications for the same. Is there a specific bit of functionality you're looking for?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#107

Post by _zoltan_ »

torginus wrote: post #103

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one. This is one of those things…

this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#108

Post by torginus »

_zoltan_ wrote: post #107
torginus wrote: post #103

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one. This is one of those things…

this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?

yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc.

Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a dollar per month for the whole bunch.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#109

Post by Dylan16807 »

texthompson wrote: post #58
danielheath wrote: post #55

Earlier quoted context omitted.

S3 supports multipart uploads which don’t necessarily send all the parts to the same server.

Why does it matter where the bytes are stored at rest? Isn't everything you need for SHA-256 just the results of the SHA-256 algorithm on every 4096-byte block? I think you could just calculate that as the data is streamed in.

> Isn't everything you need for SHA-256 just the results of the SHA-256 algorithm on every 4096-byte block?

No, you need the hash of the previous block before you can start processing the next block.

Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)

#110

Post by ncruces »

koolba wrote: post #3

This combined with the read-after-write consistency guarantee is a perfect building block (pun intended) for incremental append only storage atop an object store. It solves the biggest problem with coordinating multiple writers to a WAL.

Both this and read-after-write consistency is single object.

So coordinating writes to multiple objects still requires… creativity.

Post reply on HN