Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 240 posts

Re: S3 Strong Consistency

#11

Post by TheDong »

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

Yes.

I've observed eventual consistency with S3 prior to this change where it took on the order of several hundred milliseconds to observe an update of an existing key.

I've observed this as recently as 2 years ago.

Re: S3 Strong Consistency

#12

Post by QuinnyPig »

This was a problem with data lakes / analytics. Small inconsistencies would trash runs; that problem now goes away and removes a lot of janky half-fixes to work around the issue.

For most common use cases it's not really an issue.

Re: S3 Strong Consistency

#13

Post by johncolanduoni »

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

There used to be a behavior with 404 caching that was pretty easy to trigger: if you race requesting an object and creating it, subsequent reads could continue to 404.

Re: S3 Strong Consistency

#15

Post by rantwasp »

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

yes. i believe newer region had read after write in a while but you needed to use the right endpoint

i remember a time when if you were using the us-standard “region” and were unlucky it could take 12 hours for your objects to become visible

Re: S3 Strong Consistency

#16

Post by roseway4 »

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

Yes. Writing objects from EMR Spark to S3 buckets in a different account. Immediately setting ACLs on the new objects fail as the objects are not yet “there”. Using EMRFS Consistent View was previously the obvious solution here, but it adds significant complexity.

Re: S3 Strong Consistency

#17

Post by content_sesh »

brian_herman wrote: post #3
WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

Yeah, I never had that problem either. But I wonder how they are addressing the https://en.wikipedia.org/wiki/CAP_theorem

Sacrificing availability I reckon.

Re: S3 Strong Consistency

#18

Post by damon_c »

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat.

> You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

Re: S3 Strong Consistency

#19

Post by allengeorge »

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

Yes, we have. Specifically:

1. LIST 2. PUT 3. LIST

would trigger situations where (3) wouldn't include the object inserted in (2). This is well-known however.

Post reply on HN