Earlier quoted context omitted.
So....given CAP, which one did they give up
I thought they have implemented Optimistic locking now to coordinate concurrent writes. How does it change anything in CAP?
The C stands for Consistency.
Amazon S3 Adds Put-If-Match (Compare-and-Swap)
151–160 of 166 posts
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#152Finally. GCP has had this for a long time. Years ago I was surprised S3 didn’t.
Azure Storage has also had this for years - https://learn.microsoft.com/en-us/rest/api/storageservices/s...
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#153To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud Storage ~1 year ago. Our bet that S3 would get it in a reasonable time-frame worked out! https://turbopuffer.com/blog/turbopuffer
Interesting that what’s basically an ad is the top comment - it’s not like this is open source or anything - can’t even use it immediately (you have to apply for access). Totally proprietary. At least elasticsearch is APGL, saying nothing of open search which also supports use of S3
I mean isn't the news story itself essentially an ad?
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#154Earlier quoted context omitted.
Even better, based on queue latency instead of length
The single best metric I've found for scaling things like this is the percent of concurrent capacity that's in use. I wrote about this in a previous HN comment: https://news.ycombinator.com/item?id=41277046 Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requ…
It is a shame that comment is not a blog post!
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#155Earlier quoted context omitted.
The single best metric I've found for scaling things like this is the percent of concurrent capacity that's in use. I wrote about this in a previous HN comment: https://news.ycombinator.com/item?id=41277046 Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requ…
It is a shame that comment is not a blog post!
You will like the Strange Loop 2017 talk about this subject:
"Stop Rate Limiting! Capacity Management Done Right" by Jon Moore https://www.youtube.com/watch?v=m64SWl9bfvk
Concurrent capacity might not be the best metric.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#156Earlier quoted context omitted.
MD5 hash collisions are unlikely to happen at random. The defect was that you can make it happen purposefully, making it useless for security.
Sure, but theoretically you could have a system where a distributed log of user generated content is built via this CAS//MD5 primitive. A malicious actor could craft the data such that entries are dropped.
My understanding of the feature, and correct me if I'm wrong, is that you are not granted write access based on a hash. You already have write access. You can use the hash to avoid overwriting someone else's data that was appended to the file in between you checking the file and writing to it. If you already have write access, the hash is irrelevant. As a bad actor, you can corrupt the data without it.
MD5 should not be used for anything security related. Granting write access based on an MD5 hash would be a huge no-no.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#157Earlier quoted context omitted.
Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…
If parts are aligned on a 1024-byte boundary and you know each part's start offset, it should be possible to use the internals of a BLAKE3 tree to get the final hash of all the parts together even as they're uploaded separately. https://github.com/C2SP/C2SP/blob/main/BLAKE3.md#13-tree-has... Edit: This is actually already implemented in the Bao project which exploits the structure of the BLAKE3 merkle tree structure…
That is very neat! I love clever uses of data structures like this.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#158Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#159Earlier quoted context omitted.
Sure, but theoretically you could have a system where a distributed log of user generated content is built via this CAS//MD5 primitive. A malicious actor could craft the data such that entries are dropped.
My understanding of the feature, and correct me if I'm wrong, is that you are not granted write access based on a hash. You already have write access. You can use the hash to avoid overwriting someone else's data that was appended to the file in between you checking the file and writing to it. If you already have write access, the hash is irrelevant. As a bad actor, you can corrupt the data without it. MD5 should not…
Right, the issue comes when a trusted writer is logging data that is sourced from an untrusted party.
Imagine a transaction log being a blob per-customer with many lines corresponding to price, sku, etc, that additionally have some “memo” field provided by the customer. A trusted distributed worker process is responsible for taking incoming requests by the user, pulling their blob down, appending the line based on the request, and CAS’ing it back in (retrying on failure). With enough effort, a particularly devious user could issue many requests with ‘memo’s engineered to not alter the MD5 of their log. This would cause some lines to be lost. An audit of their account transaction log would be unable to accurately reflect the requests they made to the service, and the failure would be invisible.
This is obviously a bit contrived – I’ll be the first to admit. But if the incentives were to exist for this to be worth someone’s time for some system, I think it would be likely to see it come up eventually.
Re: Amazon S3 Adds Put-If-Match (Compare-and-Swap)
#160Earlier quoted context omitted.
Even better, based on queue latency instead of length
The single best metric I've found for scaling things like this is the percent of concurrent capacity that's in use. I wrote about this in a previous HN comment: https://news.ycombinator.com/item?id=41277046 Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requ…
Yeah this is why I hate AWS - I did a similar task runner thing and what I ended up doing is just firing up a small controller instance which manually creates and destroys instances based on demand, and schedules work on them by ssh-ing into the running instances, and piping the logs to a db.
I did read up on the 'proper' solution and it made my head spin.
You're supposed to use AWS batch, creating instances with autoscaling groups, pipe the logs to CloudWatch, and serve it from the on the frontend etc.
The number of new concepts I'd have to master, I have no control over if they went wrong, except to chase after internet erudites and spending weeks talking to AWS support is staggering.
And there's the little things, like CloudWatch logs costing like $0.5/GB, while an EBS block volume costs like $0.08, with S3 being even cheaper than that.
If I go full AWS word salad, I'm pretty sure even the most wizened AWS sages would have no idea what my bills would look like.
Yeah, my solution is shit and Im a filthy subhuman, but at least I know how every part of my code works, and the amount of code I'd had to write is not more than double that if I used AWS solutions, but I probably saved a lot of time debugging proprietary infra.