Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 240 posts

Re: S3 Strong Consistency

#61

Post by HiJon89 »

yodsanklai wrote: post #38

Could someone describe a few real-life scenarios where this is useful and noticeable?

We used to use S3 for Maven artifact storage. This is mostly an append-only workload, however Maven updates maven-metadata.xml files in place. These files contain info about what versions exist, when they were updated, what the latest snapshot version is, etc. We would see issues where a Maven build publishes to S3, and then a downstream build would read an out-of-date maven-metadata.xml and blow up. Or worse, it could silently use an older build and cause a nasty surprise when you deploy. It only happened a small percentage of the time, but when you’re doing tens of thousands of builds per day it ends up happening every day.

We switched to GCS for our Maven artifacts and the problem went away.

Re: S3 Strong Consistency

#62

Post by gaul »

dheera wrote: post #23
damon_c wrote: post #18

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.

This is not expected behavior. An older version had pessimized locking which preventing concurrently adding a file to a directory and listing the directory simultaneously which is the most similar symptom. I recommend upgrading to the latest version, running with debug flags `-f -d -o curldbg`, and observing what's happening. Please open an issue at https://github.com/s3fs-fuse/s3fs-fuse/issues if your symptoms persist.

Re: S3 Strong Consistency

#64

Post by howlgarnish »

k__ wrote: post #55

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)

Re: S3 Strong Consistency

#66

Post by basicneo »

boulos wrote: post #53

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now…

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop

Is this public?

Re: S3 Strong Consistency

#67

Post by ceocoder »

basicneo wrote: post #66
boulos wrote: post #53

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now…

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop Is this public?

Yes, https://github.com/GoogleCloudDataproc/hadoop-connectors

disclosure: I work at Google as well.

Re: S3 Strong Consistency

#68

Post by gopalv »

boulos wrote: post #53

Disclosure: I work on Google Cloud. This is super awesome for customers. I am also beyond excited for all the open-source connectors to finally be simplified so that they don't have to deal with the "oh right, gotta be careful because of list consistency". It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors. I'm now…

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop

It's been a few years, but I lost customer data on gs:// due to listing consistency - create a bunch of files, list dir, rename them one by one to commit the dir into Hive - listing missed a file & the rename skipped the missing one.

Put a breakpoint and the bug disappears, hit the rename and listing from the same JVM, the bug disappears.

Consistency issues are hell.

> It was super awesome when we were able to delete a huge chunk of the GCS Connector for Hadoop, and I hope to see the same across the S3-focused connectors.

Yes!

S3guard was a complete pain to secure, since it was built on top of dynamodb and since any client could be a writer for some file (any file), you had to give all clients write access to the entire dynamodb table, which was quite a security headache.

Re: S3 Strong Consistency

#69

Post by gaul »

banana_giraffe wrote: post #39
dheera wrote: post #23

Earlier quoted context omitted.

Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.

One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's al…

Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.

Re: S3 Strong Consistency

#70

Post by crazyjncsu »

howlgarnish wrote: post #64
k__ wrote: post #55

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

Are there any scenarios where you would want eventual consistency over strong consistency? (Assuming pricing, performance, replication etc are the same.)

No.
Post reply on HN