Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

91–100 of 240 posts

Re: S3 Strong Consistency

#91

Post by yftsui »

jz-amz wrote: post #41

Some related reading - * Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... * Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaborat...

I am curious about how much Dropbox pays for data ingress/egress, they migrated storage to their own on premise data center, then now moving data back to S3 for the data lake.

Re: S3 Strong Consistency

#93

Post by acdha »

howlgarnish wrote: post #64
k__ wrote: post #55

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)

Your last sentence is the answer I’ve always heard given: this in the context of a performance trade off — do you wait for every node in the system to acknowledge an update and purge caches, or return as soon as a sufficient durability threshold has been hit?

Re: S3 Strong Consistency

#95

Post by nullsense »

nhumrich wrote: post #79
howlgarnish wrote: post #64

Earlier quoted context omitted.

Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)

At the benefit of added availability? absolutely. This is the entire premise of mongodb.

MongoDB is webscale.

Re: S3 Strong Consistency

#97

Post by foxhop »

FridgeSeal wrote: post #86
staticassertion wrote: post #60

Earlier quoted context omitted.

Like RedShift? AWS uses S3 as a database, seems reasonable that their customers would.

That doesn't mean it blindly "mounts" it as a disk and attempts to use it as a filesystem. They're almost certainly using it in a manner that is aware of the specificities of how S3 works.

What if the blind mount works 99.9% of the time? I knew what I was doing was risky but well with in my tolerance for pain. Now that we have a strong consistency my code is fixed. That's pretty impressive.

Re: S3 Strong Consistency

#98

Post by ChuckMcM »

This is awesome, I could have used it for a project we did a while back.

I'm sure that somewhere there is a Product Manager at Amazon who talks to customers they find out are using Google Cloud or Azure and asks them "Why not use AWS?" and they mumble some feature of the other service that they need.

Sometime later, said service shows up as part of AWS. :-)

For me this is the best part of tech rivalries, whether they are in CPU performance/architecture, programming language features/tools, or network services. Pressure to improve.

Re: S3 Strong Consistency

#99

Post by basicneo »

boulos wrote: post #81
ceocoder wrote: post #67

Earlier quoted context omitted.

Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors disclosure: I work at Google as well.

Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S…

Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.
Post reply on HN