Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Strong Consistency

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 240 posts

Re: S3 Strong Consistency

#101

Post by Scaevolus »

yftsui wrote: post #91
jz-amz wrote: post #41

Some related reading - * Jeff Barr's blog post on this topic, has an interesting case study: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea... * Collaboration with Hadoop/S3A maintainers leading to this release: https://aws.amazon.com/blogs/opensource/community-collaborat...

I am curious about how much Dropbox pays for data ingress/egress, they migrated storage to their own on premise data center, then now moving data back to S3 for the data lake.

That's analytics data, not file storage. If they have 34PB of analytics, they certainly have far more than that of blob storage.

Re: S3 Strong Consistency

#102

Post by sudeepj »

whateveracct wrote: post #94

This is really something, especially for small apps. S3 is now the easiest KV store I can imagine.

> small apps

Even the big ones. A lot of applications (e.g backup vendors) had to resort to use EBS volumes because of lack of read-after-update consistency.

The barrier to entry for lot of developers has been reduced.

My prediction: A lot of storage infra (FS, DB) with native support for S3 will get commoditised now.

Re: S3 Strong Consistency

#103

Post by Decker87 »

FridgeSeal wrote: post #40
QuinnyPig wrote: post #12

This was a problem with data lakes / analytics. Small inconsistencies would trash runs; that problem now goes away and removes a lot of janky half-fixes to work around the issue. For most common use cases it's not really an issue.

Of course the real issue there is those tool treating S3 as something it's not (a filesystem) and building things on it with the expectation of certain guarantees that it explicitly didn't provide before now. I dig the feature (strong consistency) but in some ways it just enables tools that were abusing it to just do so more easily.

Isn't that the point? To provide some useful functionality for others to build on (and pay for)?

Re: S3 Strong Consistency

#104

Post by [deleted] »

basicneo wrote: post #99
boulos wrote: post #81

Earlier quoted context omitted.

Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S…

Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.

[deleted]

Re: S3 Strong Consistency

#105

Post by ultimoo »

> S3 consistency is available at no additional cost and removes the need for additional third-party, services, and complex architecture.

I wonder whether any companies/businesses solely depended on offering eventual consistency workarounds that probably now need to pivot.

Re: S3 Strong Consistency

#106

Post by banana_giraffe »

gaul wrote: post #69
banana_giraffe wrote: post #39

Earlier quoted context omitted.

One thing I noticed when I went digging through the s3fs source trying to answer that myself is that it's uploader is single threaded. Part of what makes AWS's cli reasonably fast is that it uses several threads to make sure to use as much bandwidth as possible. If I limit AWS's tool to one thread on a box with plenty of bandwidth, it turns my upload down from around 135 MiB/s to around 20 MiB/s Not sure if that's al…

Could you expand on this comment? s3fs uses multiple threads for uploading and populating readdir metadata via S3fsMultiCurl::MultiPerform. Earlier versions used curl_multi_perform which may have hidden some of the parallelism from you.

Huh, right you are. I did a cursory search, but whatever I saw clearly was wrong.

That said, something is going on that I can't explain:

    # Default settings for aws
    $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/a
    upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/a

    real    0m10.312s
    user    0m5.909s
    sys     0m4.204s

    $ aws configure set default.s3.max_concurrent_requests 4
    $ time aws s3 cp /dev/shm/1GB.bin s3://test-kihaqtowex/b
    upload: ../../dev/shm/1GB.bin to s3://test-kihaqtowex/b

    real    0m26.732s
    user    0m4.989s
    sys     0m2.741s

    $ time cp /dev/shm/1GB.bin s3fs_mount_-kihaqtowex/c

    real    0m26.368s
    user    0m0.006s
    sys     0m0.699s
(I did multiple runs, though not hundreds so I can't say with certainty this would hold)

I swear the difference was worse in the past, so maybe things have improved. Still not worth the added slowness to me for my use cases.

Re: S3 Strong Consistency

#107

Post by boulos »

basicneo wrote: post #99
boulos wrote: post #81

Earlier quoted context omitted.

Right, https://github.com/GoogleCloudDataproc/hadoop-connectors/rel... was apparently the release: > Delete metadata cache functionality because Cloud Storage has strong native list operation consistency already. If folks are actually interested in these connectors, I'd also recommend this blogpost from last year: https://cloud.google.com/blog/products/data-analytics/new-re... because even with consistency, GCS and S…

Do you have a link to the commits that removed the code. It'd be good to see what sort of complexity this sort of strong consistency can make redundant.

(Phone reply, sorry for the brevity)

Just comparing to the previous release:

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

there are some big deletions like in:

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

and

https://github.com/GoogleCloudDataproc/hadoop-connectors/com...

Igor would know more :)

Re: S3 Strong Consistency

#108

Post by deathanatos »

k__ wrote: post #55

It's interesting to read all these comments here that talk about the eventual consistency like it was some kind of bug.

The previous behavior (new keys weren't strongly consistent) were weird, and made a lot of patterns that AWS itself sort of guides you towards, broken.

For example, you can set up an SQS queue to be informed of writes into a bucket. The idea being that some component writes into the bucket, and a consumer then fetches the data & processes it. Except — gotcha! — there wasn't a guarantee of consistency. It would usually work, though, and so you'd get these weird failures that you couldn't explain unless you already knew the answer, really.

Same thing with how S3 object names sort of feel like file names, and that you might use them for something useful, and you design around that, and then you hit the section of the docs that says "no, you shouldn't, they should be uniformly balanced". (Though honestly, I've never felt any negative side-effects from ignoring this, unlike the consistency guarantees.)

Re: S3 Strong Consistency

#109

Post by [deleted] »

dheera wrote: post #23
damon_c wrote: post #18

The more I learn about S3 the more I feel the same awe I feel when looking at the Pyramids or Ankor Wat. > You can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket.

Side question -- why is s3fs so slow? I get only about 1 put per 10 seconds with s3fs. One would hope that you could use something like s3fs as a makeshift S3 API, but its performance is really horrible.

[deleted]
Post reply on HN