HopsFS: 100x Times Faster Than AWS S3

We haven't run mv/list experiments on 100,000 and 1,000,000 files for the blog. However, We expect the gap in latency between HopsFS/S3 and EMRFS would increase even further with larger directories. In our original HopsFS paper, we showed that the latency for mv/rename for a directory with 1 million files was around 5.8 seconds.

Re: HopsFS: 100x Times Faster Than AWS S3

#5

Post by Lucasoato » Sat, Nov 21, 2020, 8:47 PM UTC

Can this be installed on AWS EMR clusters?

Re: HopsFS: 100x Times Faster Than AWS S3

#6

Post by fwip » Sat, Nov 21, 2020, 8:49 PM UTC

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

What tradeoffs did you make? In what situations does S3 have better characteristics than HopsFS?

Re: HopsFS: 100x Times Faster Than AWS S3

#7

Post by fh973 » Sat, Nov 21, 2020, 8:51 PM UTC

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

> ... but has 100X the performance of S3 for file move/rename operations

Isn't rename in S3 effectively a copy-delete operation?

Re: HopsFS: 100x Times Faster Than AWS S3

#8

Post by de_Selby » Sat, Nov 21, 2020, 8:52 PM UTC

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

How does this differ from Objectivefs?

Re: HopsFS: 100x Times Faster Than AWS S3

#9

Post by dividuum » Sat, Nov 21, 2020, 8:56 PM UTC

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

The linked post says "has the same cost as S3", yet on the linked pricing page there's only a "Contact Us" Enterprise plan besides a free one. Am I missing something?

Re: HopsFS: 100x Times Faster Than AWS S3

#10

Post by [deleted] » Sat, Nov 21, 2020, 9:07 PM UTC

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

[deleted]