finally
Amazon S3 Adds Put-If-Match (Compare-and-Swap)
131–140 of 166 posts
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#132Earlier quoted context omitted.
It's right there: > Our bet that S3 would get it in a reasonable time-frame worked out!
How? This is a technical forum. Unless you’re saying any consumer of S3 can now spam links to their product on this thread with impunity. (Hey maybe they’re using cas).
Oh look, someone is mad on the internet about something silly.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#133Earlier quoted context omitted.
If you use hourly billed machines...Sounds like the world most expensive semaphore :-)
EC2 bills by the second.
Some...
"Your Amazon EC2 usage is calculated by either the hour or the second based on the size of the instance, operating system, and the AWS Region where the instances are launched" - https://repost.aws/knowledge-center/ec2-instance-hour-billin...
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#134Earlier quoted context omitted.
this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?
yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a doll…
> we spend less than a dollar per month for the whole bunch
This does not change the point, I'm just being pedantic, but:
4GB of gp3 EBS takes $0.32 per month, assuming a 50% discount (not unusual), less than a dollar gives only... 6 instances.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#135I had no idea people rely on S3 beyond dumb storage. It almost feels like people are trying to build out a distributed OLAP database in the reverse direction.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#136Earlier quoted context omitted.
If you use hourly billed machines...Sounds like the world most expensive semaphore :-)
except we are actually using them :)
Just don't call them before the hour and start a different one again.Because otherwise within the hour, you will be billed for hundreds of hours...If they are of the type billed by the hour....
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#137To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer
Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3
I don't mind hearing another developer's use case for this feature, even if it's commercial proprietary software.
It's no longer top comment, which is fine.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#138Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one. This is one of those things…
What would you say would be the "clean" way to implement a pool of workers (using EC2 instances too)?
Autoscaling and task queue based workloads, if my cloud theory is still relevant.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#139It's also possible to enforce the use of conditional writes: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... My biggest wishlist item for S3 is the ability to enforce that an object is named with a name that matches its hash. (With a modern hash considered secure, not MD5 or SHA1, though it isn't supported for those either.) That would make it much easier to build content-addressible storage.
While it can't be done server-side, this can be done straightforwardly in a signer service, and the signer doesn't need to interact with the payloads being uploaded. In other words, a tiny signer can act as a control plane for massive quantities of uploaded data. The client sends the request headers (including the x-amz-content-sha256 header) to the signer, and the signer responds with a valid S3 PUT request (minus b…
That's incredibly interesting, thank you! That's a really creative approach, and it looks like it might work for me.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#140Earlier quoted context omitted.
What would you say would be the "clean" way to implement a pool of workers (using EC2 instances too)?
Autoscaling and task queue based workloads, if my cloud theory is still relevant.
Agreed. Scaling based on the length of the queue, up to some maximum.