Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

131–140 of 240 posts

Re: S3 Strong Consistency

#131

Post by iblaine »

AWS Re:invent talked about improvements to S3 earlier today. You can see that talk here https://virtual.awsevents.com/media/1_usujusng

For any AWS EMR/hadoop users out there, this means the end of emrfs.

For the uninitiated, emrfs recognizes consistency problems but does not fix them. It'll throw an exception if there's a consistency problem, even give you some tools to fix the problem, but the tools may give false positives. The result is you've got to fix some consistency problems yourself, parse items out of the emrfs DynamoDB catalog, match them up to s3, then make adjustments where needed. It's an unnecessary chore.

It surprises me that this issue has not got more attention over the years and thankfully it'll be solved soon.

Re: S3 Strong Consistency

#132

Post by bboreham »

yftsui wrote: post #91
jz-amz wrote: post #41

Some related reading - * Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... * Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaborat...

I am curious about how much Dropbox pays for data ingress/egress, they migrated storage to their own on premise data center, then now moving data back to S3 for the data lake.

Amazon has hard-disk products for large transfers. Snowcone, Snowball and Snowmobile.

Re: S3 Strong Consistency

#134

Post by randallsquared »

web007 wrote: post #122
damon_c wrote: post #18

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

One fun fact that I learned recently: a prefix is not strictly path-delimited. I would think of /foo/bar and /foo/baz and /bar/baz as having two prefixes, but it could be anywhere from one to three, depending on how S3 has partitioned your data.

It shouldn't be one if you're sending 3500+ requests a second. It may take some time for partitioning to happen, though.

Re: S3 Strong Consistency

#135

Post by gigatexal »

boulos wrote: post #76
gopalv wrote: post #68

Earlier quoted context omitted.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one. Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the…

If it's been a few years, it was likely before the migration to Spanner: https://cloud.google.com/blog/products/gcp/how-google-cloud-... We posted that blog in early 2018, but stated: > Last year we migrated all of Cloud Storage metadata to Spanner, Google’s globally distributed and strongly consistent relational database. So maybe pre-2017?

Had no idea this happened. Very cool to see. We use both GCS and S3.

Re: S3 Strong Consistency

#136

Post by waynesonfire »

ceocoder wrote: post #67
basicneo wrote: post #66

Earlier quoted context omitted.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?

Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.

thanks for the disclosure!!! nobody cares.

Re: S3 Strong Consistency

#137

Post by jabl »

marwan-nwh wrote: post #130
danenania wrote: post #120

Earlier quoted context omitted.

The CAP theorem doesn't preclude strong consistency. It just means that at a certain scale, you'll pay for it with extra latency.

And that latency means that some reads won't see the latest write, right? Do I understand this correctly?

If you want consistency, you have to pay for it with latency, since the system has to do work in the background to ensure that the data you read is really the latest.

If you care more about latency than consistency you have an 'eventually consistent' system, where a write will eventually propagate, but a read might get stale data.

Re: S3 Strong Consistency

#139

Post by tutfbhuf »

Ask HN: I have a question regarding strong consistency. The question is related to all geo distributed strong consistent storages, but let me pick Google Spanner as example.

Spanner has a maximum upper bound of 7 milliseconds clock offset between nodes, thx to TrueTime. To achieve external consistency, Spanner simply waits 7 ms before committing a transaction. After that amount of time the window of uncertainty is over and we can be sure, that every transaction happens to be in the right order. So far so good, but how does Spanner deal with cross data center network latency? Let's say I commit a transaction to Spanner in Canada and after 7 ms, I get my confirmation, but now in Australia someone also does a conflicting transaction and also get his confirmation after 7 ms. Spanner however, bound to the laws of physics, can only see this conflict after 100+ ms delay, due to network latency between the datacenters. How is that problem solved?

Post reply on HN