Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HopsFS: 100x Times Faster Than AWS S3

logicalclocks.com

Post reply on HN
Print view View original

121–130 of 139 posts

Re: HopsFS: 100x Times Faster Than AWS S3

#121

Post by dividuum »

Dunedan wrote: post #116
dividuum wrote: post #108

Earlier quoted context omitted.

> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, o…

> For new objects, you get read-after-write consistency and the example you gave contradicts that. Mind the documented caveat for this case: > The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.

Indeed, although that seems irrelevant for the mentioned use case. After all, why would the client request an image before it's uploaded. It seems like a straight forward "server side upload->generate url and send it to client->client requests image" flow. It's perfectly covered by the consistency provided.

Re: HopsFS: 100x Times Faster Than AWS S3

#122

Post by fxtentacle »

dividuum wrote: post #108
fxtentacle wrote: post #103

Earlier quoted context omitted.

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our su…

> S3 requests to US buckets might be served by delayed replicas in other countries if the request comes from outside the US What? That makes no sense. Do you have a source for that? I thought the explicit choice of region when creating a bucket limits where data is located. Why would the give you geo-replication for free? Also: "Amazon S3 creates buckets in a Region you specify. To optimize latency, minimize costs, o…

They also say "A process writes a new object to Amazon S3 and immediately lists keys within its bucket. Until the change is fully propagated, the object might not appear in the list" which implies that you cannot read immediately after write.

Re: HopsFS: 100x Times Faster Than AWS S3

#123

Post by fxtentacle »

Galanwe wrote: post #107
fxtentacle wrote: post #103

Earlier quoted context omitted.

> write your app against S3 and call it a day. ... and then later you notice that you need a special SLA and Amazon Redshift to guarantee that a read from S3 will return the same value as the last write. Even S3 is only eventually consistent and especially if a US user uploads images into your US bucket and then you serve the URLs to EU users, you might have loading problems. The correct solution, according to our su…

> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. No…

Depending on your write throughput, versioning can be quite expensive.

Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.

Re: HopsFS: 100x Times Faster Than AWS S3

#124

Post by fxtentacle »

dividuum wrote: post #121
Dunedan wrote: post #116

Earlier quoted context omitted.

> For new objects, you get read-after-write consistency and the example you gave contradicts that. Mind the documented caveat for this case: > The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.

Indeed, although that seems irrelevant for the mentioned use case. After all, why would the client request an image before it's uploaded. It seems like a straight forward "server side upload->generate url and send it to client->client requests image" flow. It's perfectly covered by the consistency provided.

That's what we did. Server side upload, link into websocket, client downloads image. It did work 99.99% of all requests.

Re: HopsFS: 100x Times Faster Than AWS S3

#125

Post by tw04 »

SteveNuts wrote: post #88
tw04 wrote: post #76

Earlier quoted context omitted.

https://www.netapp.com/cloud-services/cloud-volumes-service-... You could try NetApp - it's significantly faster than EFS in my experience.

This solution is at least 40k per year for just the disk controllers, so be prepared for that.

First off we're talking about AWS, not on-premises. Second, if we were talking on-premises, you can get into a NetApp controller for FAR less than $40k a year, but that's not really relevant to OP.

There are two options in AWS: one where you get an entire virtual storage system to yourself but also need to have some NetApp expertise (although less than on-premises). You can find pricing here:

https://cloud.netapp.com/pricing

And one where you just consume the NFS/SMB shares more akin to EFS (pricing at the bottom of the page):

https://cloud.netapp.com/cloud-volumes-service-for-aws

In both instances you can get better pricing than what is on that page by working with a partner, but if you don't want to you can buy directly through AWS and pay a slight premium.

Re: HopsFS: 100x Times Faster Than AWS S3

#126

Post by daviesliu »

We see similar results using JuiceFS [1], are glad to see that another player has moved on the same direction with us, since JuiceFS was released to public in 2017, and is available on most of public cloud vendor.

The architect of JuiceFS is simper than HopsFS, it does not have worker nodes (the client access S3 and manage cache directly), and the metadata is stored in highly optimized server (similar to NN in HDFS, can be scaled out by adding more nodes).

JuiceFS provide POSIX client (using FUSE), and Hadoop SDK (in Java), and a S3 gateway (also a WebDAV gateway).

[1]: https://juicefs.com/docs/en/metadata_performance_comparison....

Disclaimer: Founder of JuiceFS here

Re: HopsFS: 100x Times Faster Than AWS S3

#127

Post by Dunedan »

fxtentacle wrote: post #124
dividuum wrote: post #121

Earlier quoted context omitted.

Indeed, although that seems irrelevant for the mentioned use case. After all, why would the client request an image before it's uploaded. It seems like a straight forward "server side upload->generate url and send it to client->client requests image" flow. It's perfectly covered by the consistency provided.

That's what we did. Server side upload, link into websocket, client downloads image. It did work 99.99% of all requests.

I'm surprised about the conclusion of the AWS support that this is intended behavior. I'm curious about the specifics of the behavior of S3 in this case. Do you have any more detailed information from the AWS support you could share by any chance? If you don't mind, I'd be also grateful if you could drop some answers to the questions below.

Do you use S3 replication (CRR/SRR)?

Before uploading, do you make any request to the object key? If yes, what kind of request?

For downloading, do you use pre-signed URLs or is the S3 bucket public?

What's the error the 0,01% of users encounter? Is it a NoSuchKey error?

Re: HopsFS: 100x Times Faster Than AWS S3

#128

Post by daviesliu »

Lucasoato wrote: post #5

Can this be installed on AWS EMR clusters?

JuiceFS can used in AWS EMR cluster, we have customers using JuiceFS to address the consistency and performance issues from S3.

There is a blog post [1] talking about this use case. Unfortunately, we have not publish a English version, you can read it using google translate .

[1] https://juicefs.com/blog/cn/posts/globalegrow-big-data-platf...

Disclaimer: Founder of JuiceFS here.

Re: HopsFS: 100x Times Faster Than AWS S3

#129

Post by daviesliu »

mwcampbell wrote: post #29

> But, until today, there has been no equivalent to ADLS for S3. ObjectiveFS has been around for several years. How is HopsFS better?

There are different when they are read/write from multiple clients in same time. HopsFS can use the meta service to synchronize with each other under low latency (about ms), but ObjectiveFS have to use S3 to do the synchronisation, which has much higher latency (> 20ms).

We have chatted on these with the founders of ObjectiveFS before creating JuiceFS, they did NOT recommend to use ObjectiveFS in big data workload with Hadoop/Spark, that's why we started to build JuiceFS since 2016.

Re: HopsFS: 100x Times Faster Than AWS S3

#130

Post by Galanwe »

fxtentacle wrote: post #123
Galanwe wrote: post #107

Earlier quoted context omitted.

> The correct solution, according to our support contact, is that we wait a second after uploading to S3 I'm shocked that you got this answer. This is definitely not how you are supposed to operate. If you need to ensure the sequantiality of a write followed by a read on S3, the idiomatic way is to enable versioning on your bucket, issue a write, and provide the version ID to whoever need to read after that write. No…

Depending on your write throughput, versioning can be quite expensive. Also, we were already at a throughput where the s3 metadata nodes would sometimes go offline, so I'm not sure putting more work on them would have improved our overall situation.

Interesting, do you have more information on how you did monitors these S3 metadata nodes? Never heard of that.

Why would an unversioned put be less costly than a versioned put? To me it should be pretty much the same. I would almost suspect unversioned puts are hacked around versioned ones.

Out of curiosity what kind of workload did you perform? I have been moving terabytes of files (some very small 1kB files, some huge 150GB files) and never noticed a single blip of change in behavior from S3 (and certainly not anything going "off")

Post reply on HN