Is Dropbox still using AWS? They had a testimonial on the page.
S3 Strong Consistency
181–190 of 240 posts
Re: S3 Strong Consistency
#182Earlier quoted context omitted.
I shall print this..
Please support the original artist and make sure you have a license to print it before you do.I
the artist is Pascal Jousselin
http://pjousselin.free.fr/
it's called 'Imbattable' or Mister Invincible
Re: S3 Strong Consistency
#183I thought we had worked out in 2010-2012 that the CAP theorem, consistency, availability and partition tolerance ( https://en.wikipedia.org/wiki/CAP_theorem#History ) made this impossible? Where is the mistake.
S3 is almost certainly not fully partition tolerant at the node level and requires some sort of quorum. Other “magical” data stores like Spanner also retain this limitation, they just have very reliable replication strategies.
Re: S3 Strong Consistency
#184Earlier quoted context omitted.
One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but it could be anywhere from one to three, depending on how S3 has partitioned your data.
It shouldn't be one if you're sending 3500+ requests a second. It may take some time for partitioning to happen, though.
This is a major issue when turning batch data into individual files, especially with dynamic prefixes. You can and will find this edge pretty quick using EMR or Glue which tries to run as fast as possible. The only answer is....slow down your writes or just try again, a bit frustrating.
Re: S3 Strong Consistency
#185One capability left to implement now: make `move` operations in S3 O(1)
Re: S3 Strong Consistency
#186Earlier quoted context omitted.
Previously, update-then-read was not guaranteed to be consistent. Now it is consistent.
yes for aws it was an issue, what i meant is that aws solved an issue that it itself created.
14 years ago. That is called tech debt. Not an issue. They designed it for the use case at the time. Nobody knew what the cloud was becoming.
Re: S3 Strong Consistency
#187Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now…
[deleted]
Re: S3 Strong Consistency
#188Earlier quoted context omitted.
Could you not already do that by specifying ETag?
No, because the object may not be there (but already uploaded) with eventual consistency.
Before this update, S3 provided "read-after-write" consistency for object creation at a unique path[0] with PUT and subsequent GET operations. Replacement wasn't consistent. So I think you could have worked out some kind of atomic pattern with S3 objects, using new keys for each state change.
[0] https://github.com/awsdocs/amazon-s3-developer-guide/blob/41...
Re: S3 Strong Consistency
#189so many posts about amazon,
i like AWS but this looks pretty much like advertising haha
Re: S3 Strong Consistency
#190Could someone describe a few real-life scenarios where this is useful and noticeable?
Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).
Same issues updating a file - you’d update on s3, then trigger processing and even though it was 50ms later - you’d get OLD version. I didn’t know gcp didn’t have this issue - would have been a big selling point