Diving Deep on S3 Consistency • Hacker News Board

Re: Diving Deep on S3 Consistency

#11

Post by valenterry » Wed, Apr 28, 2021, 1:47 PM UTC

Here's what I take away from this post:

> We built automation that can respond rapidly to load concentration and individual server failure. Because the consistency witness tracks minimal state and only in-memory, we are able to replace them quickly without waiting for lengthy state transfers.

So this means that the "system" that contains the witness(es) is a single point of truth and failure (otherwise we would lose consistency again), but because it does not have to store a lot of information, it can be kept in-memory and can be exchanged quickly in case of failure.

Or in other words: minimize the amount of information that is strictly necessary to keep a system consistent and then make that part its own in-memory and quickly failover-able system which is then the bar for the HA component.

Is that what they did?

Re: Diving Deep on S3 Consistency

#12

Post by jeffbarr » Wed, Apr 28, 2021, 1:54 PM UTC

crashocaster wrote: post #9

I would have been interested to hear more about the verification techniques and tools they used for this project.

Check out https://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-... ("How Amazon Web Services Uses Formal Methods") and https://d1.awsstatic.com/Security/pdfs/One_Click_Formal_Meth... ("One-Click Formal Methods") for more info.

Re: Diving Deep on S3 Consistency

#13

Post by iblaine » Wed, Apr 28, 2021, 1:59 PM UTC

Anyone else still seeing consistency problems w/S3 & EMR? The latest AWS re:Invent made it sound like this would be fixed but as of yesterday I was still using emrfs to correct S3 consistency problems.

Re: Diving Deep on S3 Consistency

#14

Post by rossmohax » Wed, Apr 28, 2021, 2:08 PM UTC

Recent S3 consistency improvements are welcome, but S3 still falls behind Google GCS until they support conditional PUTs.

GCS allows object to be replaced conditionally with `x-goog-if-generation-match` header, which sometimes can be quite useful.

Re: Diving Deep on S3 Consistency

#15

Post by ignoramous » Wed, Apr 28, 2021, 2:32 PM UTC

rossmohax wrote: post #14

Recent S3 consistency improvements are welcome, but S3 still falls behind Google GCS until they support conditional PUTs. GCS allows object to be replaced conditionally with `x-goog-if-generation-match` header, which sometimes can be quite useful.

Vogels spoke briefly about why AWS prefers versioned objects instead here: https://queue.acm.org/detail.cfm?id=3434573

BTW, DynamoDB supports conditional PUTs if your data can fit under 400 KiB.

Re: Diving Deep on S3 Consistency

#16

Post by sidereal » Wed, Apr 28, 2021, 2:36 PM UTC

crashocaster wrote: post #9

I would have been interested to hear more about the verification techniques and tools they used for this project.

The folks involved gave a neat talk about the verification techniques and tools they used as part of AWS Pi Week recently: https://www.twitch.tv/videos/951537246?t=1h10m10s

Re: Diving Deep on S3 Consistency

#17

Post by pawelmi » Wed, Apr 28, 2021, 2:47 PM UTC

So it is both available and consistent (but perhaps only in read your own writes way?). What is then with resilence to network partitions, referring to CAP theorm? Did they build super reliable global network, so this is never a real issue?

Re: Diving Deep on S3 Consistency

#18

Post by [deleted] » Wed, Apr 28, 2021, 2:50 PM UTC

crashocaster wrote: post #9

I would have been interested to hear more about the verification techniques and tools they used for this project.

[deleted]

Re: Diving Deep on S3 Consistency

#19

Post by swyx » Wed, Apr 28, 2021, 3:14 PM UTC

wolf550e wrote: post #10

AWS fixed S3 consistency in December 2020: https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3...

thats... what the post is about..

Re: Diving Deep on S3 Consistency

#20

Post by swyx » Wed, Apr 28, 2021, 3:24 PM UTC

i find this very light on the actual "diving deep" part promised in the title. theres a lot of self congratulatory chest thumping, not a lot of technical detail. Werner of course doesnt owe us any explanation whatsoever. i just dont find this particularly deep.