Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HopsFS: 100x Times Faster Than AWS S3

logicalclocks.com

Post reply on HN
Print view View original

101–110 of 139 posts

Re: HopsFS: 100x Times Faster Than AWS S3

#101

Post by jamesblonde »

ryanmarsh wrote: post #65
geertj wrote: post #44

Earlier quoted context omitted.

> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)

Hi, my use case for EFS Lambda is burts of small writes. Could you add a perf mode that favors bursts of small writes? We did some perf tests and just couldn’t make it fast enough.

What's a small write in your use case? How many KBs is the write op, and how many ops/sec?

We can store small files on NVMe disks in metadata, so that you can read them with a latency of just a couple of ms, and can scale to 10s of thousands of concurrent reads/writes per second with small files.

https://www.logicalclocks.com/blog/millions-and-millions-of-...

Re: HopsFS: 100x Times Faster Than AWS S3

#102

Post by nostrebored »

jamesblonde wrote: post #91
xyzzy_plugh wrote: post #72

Earlier quoted context omitted.

The parent does make a good point about centralization of requests being a problem. S3 load balances under the hood, so different key prefixes within a bucket are usually serviced in isolation -- a DoS to one prefix will usually not affect other prefixes. It seems like you'd be limiting yourself for concurrent access -- if everything is flowing through the MySQL cluster -- not a bad thing! Just perhaps warrants a cav…

Read the paper, it doesn't smoke HopsFS on concurrency. In the 2nd paper, we get 1.6m ops/sec on HopsFS HA over 3 availability zones (without S3 as a backing layer). Can you get that on S3 (maybe if you are Netflix, otherwise your starting quota is 1000s ops/sec)?

"Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second."

With 500 prefixes it doesn't matter if you're Netflix or a student working on a distributed computing project, you can get that 1.6m requests.

There is no limit to the number of prefixes in a bucket. You can scale to be arbitrarily large.

Re: HopsFS: 100x Times Faster Than AWS S3

#103

Post by fxtentacle »

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

> write your app against S3 and call it a day.

... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write.

Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our support contact, is that we wait a second after uploading to S3 to make sure at least the metadata has replicated to the EU before we post the image url. That, or pay for Redshift to tell us how long to wait more precisely.

Because contrary to what everyone expects, S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US.

Re: HopsFS: 100x Times Faster Than AWS S3

#104

Post by Proven »

SteveNuts wrote: post #88
tw04 wrote: post #76

Earlier quoted context omitted.

https://www.netapp.com/cloud-services/cloud-volumes-service-... You could try NetApp - it's significantly faster than EFS in my experience.

This solution is at least 40k per year for just the disk controllers, so be prepared for that.

The small 24 drive all flash model (dual controller) is around $20k, based on a quick google search

Re: HopsFS: 100x Times Faster Than AWS S3

#105

Post by zelly »

catlifeonmars wrote: post #71

S3 isn’t a file system. It’s a key-value store. If you’re trying to use it as a file system, you’re doing it wrong :)

Eventually someone will find a coincidental or unintended aspect of your API and then depend on it. This article is the ultimate exemplar of this rule.

Re: HopsFS: 100x Times Faster Than AWS S3

#106

Post by untech »

From the title, I though that this is a technology which gives you 100x faster static sites compared to S3, which sounded really cool. Discovering that it was about move operations was really underwhelming. The title is on the edge of being misleading.

Re: HopsFS: 100x Times Faster Than AWS S3

#107

Post by Galanwe »

fxtentacle wrote: post #103
tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our su…

> The correct solution, according to our support contact, is that we wait a second after uploading to S3

I'm shocked that you got this answer. This is definitely not how you are supposed to operate.

If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write.

Not only will that transparently ensure that you will not read deprecated data, but it will even ensure that you actually read the result of that particular write, and not any consecutive write that could have happened in between.

This pattern is very easy to implement for sequential operations in a single function, like:

    version = s3.put_object(...)
    data = s3.get_object(version_id=version, ...)
I agree that when you deal with various processes it can become messy.

In practice it never bothered me too much though. I prefer having explicit synchronization through versions, rather than having a blocking write waiting for all caches to be synchronized.

Also, this should only be necessary if you write to an already existing object. New keys will always trigger a cache invalidation from my understanding.

Re: HopsFS: 100x Times Faster Than AWS S3

#108

Post by dividuum »

fxtentacle wrote: post #103
tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our su…

> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US

What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, or address regulatory requirements, choose any AWS Region that is geographically close to you." - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket....

Also the consistency when creating objects is described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.... For new objects, you get read-after-write consistency and the example you gave contradicts that.

Re: HopsFS: 100x Times Faster Than AWS S3

#109

Post by smarx007 »

untech wrote: post #106

From the title, I though that this is a technology which gives you 100x faster static sites compared to S3, which sounded really cool. Discovering that it was about move operations was really underwhelming. The title is on the edge of being misleading.

I think the right title may have been "HopsFS: speeding up HDFS over S3 by up to 100x"

Re: HopsFS: 100x Times Faster Than AWS S3

#110

Post by jamesblonde »

nostrebored wrote: post #102
jamesblonde wrote: post #91

Earlier quoted context omitted.

Read the paper, it doesn't smoke HopsFS on concurrency. In the 2nd paper, we get 1.6m ops/sec on HopsFS HA over 3 availability zones (without S3 as a backing layer). Can you get that on S3 (maybe if you are Netflix, otherwise your starting quota is 1000s ops/sec)?

"Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could…

S3 has quotas. Go try and get 1m ops/sec as a quota.
Post reply on HN