S3 Strong Consistency • Hacker News Board

Re: S3 Strong Consistency

#81

Post by boulos » Wed, Dec 02, 2020, 2:24 AM UTC

ceocoder wrote: post #67

basicneo wrote: post #66

Earlier quoted context omitted.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?

Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.

Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release:

> Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already.

If folks are actually interested in these connectors, I'd also recommend this blogpost from last year:

https://cloud.google.com/blog/products/data-analytics/new-re...

because even with consistency, GCS and S3 still aren't filesystems :).

Re: S3 Strong Consistency

#82

Post by foota » Wed, Dec 02, 2020, 2:34 AM UTC

hoytschermerhrn wrote: post #71

kleebeesh wrote: post #47

Fantastic. Any write-ups or descriptions for how they made it happen?

Seconded, I'd love to read a whitepaper if anyone is able to provide one. I'm assuming this operates on some variant of Google's TrueTime[0]. [0] https://cloud.google.com/spanner/docs/true-time-external-con...

Not necessarily, I think Spanner's magic is most useful for bounded staleness reads and scaling reads through non-leader replicas. Otherwise I think you're looking at normal "NewSQL" guarantees.

Re: S3 Strong Consistency

#83

Post by enw » Wed, Dec 02, 2020, 2:37 AM UTC

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

All the time when using EMR with S3. Really frequent, and really annoying to mitigate.

Re: S3 Strong Consistency

#84

Post by enw » Wed, Dec 02, 2020, 2:39 AM UTC

roseway4 wrote: post #16

WrtCdEvrydy wrote: post #2

Has anyone ever seen S3 behave eventually consistent? I have not seen a lot of eventual consistency in the real world but I wonder if I'm just working on the wrong problems?

Yes. Writing objects from EMR Spark to S3 buckets in a different account. Immediately setting ACLs on the new objects fail as the objects are not yet “there”. Using EMRFS Consistent View was previously the obvious solution here, but it adds significant complexity.

EMRFS consistent view is a mess. Changes to S3 files are likely to break EMRFS, the metadata needs to be kept in sync (otherwise the table will keep increasing forever) with tools that are a pain to install, and when I used it to run Hive queries from a script, it didn't work.

Re: S3 Strong Consistency

#85

Post by stubish » Wed, Dec 02, 2020, 2:40 AM UTC

Once libraries and tools start relying on this, it is going to make life interesting to the S3-compatible players. The API remains the same, but the behavior is quite different.

Re: S3 Strong Consistency

#86

Post by FridgeSeal » Wed, Dec 02, 2020, 2:43 AM UTC

staticassertion wrote: post #60

FridgeSeal wrote: post #40

Earlier quoted context omitted.

Of course the real issue there is those tool treating S3 as something it's not (a filesystem) and building things on it with the expectation of certain guarantees that it explicitly didn't provide before now. I dig the feature (strong consistency) but in some ways it just enables tools that were abusing it to just do so more easily.

Like RedShift? AWS uses S3 as a database, seems reasonable that their customers would.

That doesn't mean it blindly "mounts" it as a disk and attempts to use it as a filesystem. They're almost certainly using it in a manner that is aware of the specificities of how S3 works.

Re: S3 Strong Consistency

#87

Post by gregw2 » Wed, Dec 02, 2020, 2:47 AM UTC

WrtCdEvrydy wrote: post #36

SpicyLemonZest wrote: post #10

Earlier quoted context omitted.

I've never seen the minutes long delays that you were theoretically supposed to accommodate, but second-scale delays were pretty common in my team's experience. (We're writing a lot of files though, probably hundreds or thousands per second aggregated across all our customers.)

Yeah, I got seconds but people were saying there were minutes of data... max I ever saw was 3 seconds on a 2gb file...

I ran across issues where files that we PUT in S3 were still not visible in Hive 6+ hours later. It only happened a few times out of hundreds of jobs per day for jobs running daily the last 2-3 years, but it was annoying to deal with when it happened because there wasn't a lot we could do beyond open tickets with AWS.

Re: S3 Strong Consistency

#88

Post by dboreham » Wed, Dec 02, 2020, 2:56 AM UTC

howlgarnish wrote: post #64

k__ wrote: post #55

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)

Train tunnels

Re: S3 Strong Consistency

#89

Post by PetahNZ » Wed, Dec 02, 2020, 2:58 AM UTC

ramraj07 wrote: post #50

PetahNZ wrote: post #42

Earlier quoted context omitted.

Our image processing worker queues/servers write an S3 object and dispatch a follow up job. Currently we have to delay the next job (we use 5 seconds) otherwise then next job may start processing before the S3 object is available (it 404's if the next job is run straight away).

That sounds _very_ wrong, what type of throughput are we talking about here?

Up to hundreds of images per second.

Re: S3 Strong Consistency

#90

Post by YZF » Wed, Dec 02, 2020, 3:02 AM UTC

cash22 wrote: post #65

How do they accomplish strong consistency without any hit to availability?

While I don't know the specifics I can say that availability is down to engineering practices.

Let's say that their consistency model is achieved via quorum, that is writes write to a quorum of nodes while reads read from a quorum of nodes (of their metadata database) then this guarantees read after write consistency.

The availability aspect of this is just engineering, making sure you're never down to less than a quorum of nodes. E.g. by having redundant power supplies, generators, networks, or whatever engineering it takes to reduce the probability of failure.

There's other aspects, e.g. latency, that may suffer, but again this is solved via engineering. Just throw more hardware at it to bring the latency down. The only time where you absolutely can't solve it is if you provide strong consistency across geographical regions that are far apart, there's no way then not to pay that latency.

This is just another example of why the CAP theorem isn't really as useful to determining limitations of practical systems as it may seem at first site.