S3 Strong Consistency
aws.amazon.com
S3 Strong Consistency
1–10 of 240 posts
Re: S3 Strong Consistency
#2Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
Re: S3 Strong Consistency
#3Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
Yeah, I never had that problem either.
But I wonder how they are addressing the
https://en.wikipedia.org/wiki/CAP_theorem
Re: S3 Strong Consistency
#4Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
I think it was already strongly consistent within regions?
like if you had tried to read a non-existent key, then wrote to it, it might continue to appear to not exist for a minute?
Re: S3 Strong Consistency
#5That's pretty incredible. With no price increase either. Using s3 as a makeshift database is possible in a lot more scenarios now.
I especially like using s3 as a db when dealing with ETLs, where the state of some data is stored in its key prefix. This means that the etl can stop / resume at any point with a single source of truth as the database.
The potential drawback is cost of course; moving (renaming) is free but copying is not. S3's biggest price pain is always its PUTs. In many ETLs this is usually a non-issue because you have to do those PUTs regardless, as you probably want the data to be saved at the various stages for future debugging and recovery.
Re: S3 Strong Consistency
#6I believe this makes Amazon S3 behave more similar to Azure blob storage[1] and Google Cloud Storage[2], which is pretty convenient for folks who are abstracting blob stores across different offerings.
For what it’s worth, consistency in S3 was usually pretty good anyways, but I ran into issues where it could vary a bit in the past. If you designed your application with this in mind, of course, it shouldn’t be an issue. In my case I believe we just had to add some retry logic. (And of course, that is no longer necessary.)
[1]: https://docs.microsoft.com/en-us/azure/storage/common/storag...
Re: S3 Strong Consistency
#7Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
I think it was already strongly consistent within regions? like if you had tried to read a non-existent key, then wrote to it, it might continue to appear to not exist for a minute?
Their previous consistency was RAW, but not RAU (for most regions. I think us-east-1 was missing RAW consistency for quite a while).
After a write, you would _always_ be able to read the key you just wrote.
After an update, you could get a stale copy of the key if your subsequent read hit a different server.
Re: S3 Strong Consistency
#8Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
Yes. When you're writing terabytes of data files it was very obvious.
That's why manifest files became so popular.
Re: S3 Strong Consistency
#9This is one reason I have been a big fan of Google Cloud Storage over AWS S3: at a past company AWS consistency was a huge pain, and GCS has had this for years.
https://cloud.google.com/blog/products/gcp/how-google-cloud-...
Re: S3 Strong Consistency
#10Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?
I've never seen the minutes long delays that you were theoretically supposed to accommodate, but second-scale delays were pretty common in my team's experience. (We're writing a lot of files though, probably hundreds or thousands per second aggregated across all our customers.)