Diving Deep on S3 Consistency • Hacker News Board

Re: Diving Deep on S3 Consistency

#41

Post by somethingAlex » Wed, Apr 28, 2021, 7:20 PM UTC

pawelmi wrote: post #17

So it is both available and consistent (but perhaps only in read your own writes way?). What is then with resilence to network partitions, referring to CAP theorm? Did they build super reliable global network, so this is never a real issue?

The consistency level seems to be Causal Consistency, which does include read-your-writes. S3 doesn't provide ACID transactions, so stricter consistency models aren't really needed.

From what I've read, if a network issue occurs which would impair consistency, S3 sacrifices availability. The write would just fail.

But this isn't your 5-node distributed system. Like they mention in the article, the witness system can remove and add nodes very quickly and it's highly redundant. A network issue that would actually cause split-brain or make it difficult to reach consensus would be few and far between.

Re: Diving Deep on S3 Consistency

#42

Post by [deleted] » Wed, Apr 28, 2021, 7:31 PM UTC

rossmohax wrote: post #14

Recent S3 consistency improvements are welcome, but S3 still falls behind Google GCS until they support conditional PUTs. GCS allows object to be replaced conditionally with `x-goog-if-generation-match` header, which sometimes can be quite useful.

[deleted]

Re: Diving Deep on S3 Consistency

#43

Post by ece » Wed, Apr 28, 2021, 8:38 PM UTC

the_reformation wrote: post #31

skynet-9000 wrote: post #30

Earlier quoted context omitted.

Why would AWS provide a feature that makes additional transaction and storage charges (as well as subsequent reads to see which is the correct version) irrelevant?

S3 (and most AWS services) are extremely price elastic; i.e., the lower you make them cost, the more people use it (a la electricity.) That's why they've done stuff like drop from 50ms billing to 1ms billing, etc.

They could still offer it as a client library feature, just tell the users what kind of r/w amplification and guarantees they can expect, and it's something they could optimize later or not.

Re: Diving Deep on S3 Consistency

#44

Post by CommanderHux » Wed, Apr 28, 2021, 9:55 PM UTC

deepsun wrote: post #23

wolf550e wrote: post #10

AWS fixed S3 consistency in December 2020: https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3...

By the way, Google's GCS had it from the beginning.

Azure had it from the beginning

Re: Diving Deep on S3 Consistency

#45

Post by nhoughto » Wed, Apr 28, 2021, 10:00 PM UTC

Would love a dive (hopefully deep) into IAM, the innards of that must be some impressive wizardry. Surprising there isn't more around about the amazing technical workings of these foundational AWS products.

Re: Diving Deep on S3 Consistency

#46

Post by kondro » Wed, Apr 28, 2021, 11:22 PM UTC

CommanderHux wrote: post #44

deepsun wrote: post #23

Earlier quoted context omitted.

By the way, Google's GCS had it from the beginning.

Azure had it from the beginning

Azure doesn't let you write more than 20MB/s to an storage account.

Re: Diving Deep on S3 Consistency

#47

Post by jiggawatts » Thu, Apr 29, 2021, 12:05 AM UTC

kondro wrote: post #46

CommanderHux wrote: post #44

Earlier quoted context omitted.

Azure had it from the beginning

Azure doesn't let you write more than 20MB/s to an storage account.

What are you talking about? The default ingress rate is 10 Gbps: https://docs.microsoft.com/en-us/azure/storage/common/scalab...

Elsewhere they state maximum scalability targets of 400 Gbps egress and 80 Gbps ingress: https://docs.microsoft.com/en-us/azure/storage/files/storage...

Re: Diving Deep on S3 Consistency

#48

Post by killingtime74 » Thu, Apr 29, 2021, 12:16 AM UTC

swyx wrote: post #20

i find this very light on the actual "diving deep" part promised in the title. theres a lot of self congratulatory chest thumping, not a lot of technical detail. Werner of course doesnt owe us any explanation whatsoever. i just dont find this particularly deep.

The higher you go the more your job is marketing

Re: Diving Deep on S3 Consistency

#49

Post by killingtime74 » Thu, Apr 29, 2021, 12:20 AM UTC

nhoughto wrote: post #45

Would love a dive (hopefully deep) into IAM, the innards of that must be some impressive wizardry. Surprising there isn't more around about the amazing technical workings of these foundational AWS products.

There are if you work for them....

Re: Diving Deep on S3 Consistency

#50

Post by redis_mlc » Thu, Apr 29, 2021, 1:21 AM UTC

So ... HNers don't want "eventual consistency", they want consistency now.

Some ignoramous roasted me last week for saying the above, and that I was "a decade behind." Like usual, I was ahead because I actually know what I'm talking about.