HopsFS: 100x Times Faster Than AWS S3 • Hacker News Board

Re: HopsFS: 100x Times Faster Than AWS S3

#41

Post by SirOibaf » Sat, Nov 21, 2020, 10:41 PM UTC

mwcampbell wrote: post #29

> But, until today, there has been no equivalent to ADLS for S3. ObjectiveFS has been around for several years. How is HopsFS better?

I'd say that the main difference with ObjectiveFS are metadata operations. From the documentation of ObjectiveFS:

`doesn’t fully support regular file system semantics or consistency guarantees (e.g. atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links).`

HopsFS does provide strongly consistent metadata operations like atomic directory rename, which is essential if you are running frameworks like Apache Spark.

Re: HopsFS: 100x Times Faster Than AWS S3

#42

Post by mwcampbell » Sat, Nov 21, 2020, 10:50 PM UTC

SirOibaf wrote: post #41

mwcampbell wrote: post #29

> But, until today, there has been no equivalent to ADLS for S3. ObjectiveFS has been around for several years. How is HopsFS better?

I'd say that the main difference with ObjectiveFS are metadata operations. From the documentation of ObjectiveFS: `doesn’t fully support regular file system semantics or consistency guarantees (e.g. atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links).` HopsFS does provide strongly consistent metadata operations like atomic directory ren…

That quote from the ObjectiveFS documentation [1] is out of context. It was describing limitations in s3fs, not ObjectiveFS. My understanding is that because ObjectiveFS is a log-structured filesystem that uses S3 as underlying storage, it doesn't have those limitations.

[1]: https://objectivefs.com/faq

Re: HopsFS: 100x Times Faster Than AWS S3

#43

Post by harshaw » Sat, Nov 21, 2020, 11:10 PM UTC

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

EFS attempts to solve this problem, although the speed of light between data centers is still a minor challenge.

Re: HopsFS: 100x Times Faster Than AWS S3

#44

Post by geertj » Sat, Nov 21, 2020, 11:29 PM UTC

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

> It usually starts with NFS, which is a big huge single point of failure.

NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not.

(disclaimer: I'm a PM-T on the EFS team)

Re: HopsFS: 100x Times Faster Than AWS S3

#45

Post by slaymaker1907 » Sat, Nov 21, 2020, 11:30 PM UTC

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

You can minimize the effects of eventual consistency by distinguishing the distant replicas from the same-datacenter ones. Cross datacenter/region might be eventually consistent, but you will only see it rarely since the same-datacenter replicas are fully consistent with each other.

The main downside is that you have to pick one datacenter as the main datacenter at any given time. This can change if the old one goes down or due to business needs, but you can't have two of them as the main datacenter at once for a given dataset.

Re: HopsFS: 100x Times Faster Than AWS S3

#46

Post by jamesblonde » Sat, Nov 21, 2020, 11:49 PM UTC

hilbertseries wrote: post #26

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

Reading through your article, this solution is built on top of s3. So, moving and listing files is faster, presumably due to a new metadata system you've built for tracking files. The trade off here, is that writes must be strictly slower now than they were previously because you've added a network hop. All read and write data now flows through these workers. Which adds a point of failure, if you steam too much data…

I'm one of the co-authors. The numbers for writes are in the paper, so it is very unfair to call it an advertisement. And it is a global cache - if the block is cached somewhere, it will be used in reads.

Re: HopsFS: 100x Times Faster Than AWS S3

#47

Post by jamesblonde » Sat, Nov 21, 2020, 11:51 PM UTC

dividuum wrote: post #9

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

The linked post says "has the same cost as S3", yet on the linked pricing page there's only a "Contact Us" Enterprise plan besides a free one. Am I missing something?

There is no extra charge for HopsFS as part of the Hopsworks platform - so you only pay for what you read/write/store in S3. There is SaaS pricing public on hopsworks.ai.

Re: HopsFS: 100x Times Faster Than AWS S3

#48

Post by jamesblonde » Sat, Nov 21, 2020, 11:52 PM UTC

ajb wrote: post #21

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

You say it's "posix-like" - so what from posix had to be left out?

Random writes are not supported in the HDFS API.

Re: HopsFS: 100x Times Faster Than AWS S3

#49

Post by randallsquared » Sun, Nov 22, 2020, 12:02 AM UTC

hartator wrote: post #11

> 100X the performance of S3 for file move/rename operations I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than something you want to do on a regular basis.

Also, I presume that "move" means "update the index, held in another object or objects" (no slight! If I understand correctly, that's what most on-disk filesystems do as well). That's the only way I can imagine getting performance that much better.

Re: HopsFS: 100x Times Faster Than AWS S3

#50

Post by jamesblonde » Sun, Nov 22, 2020, 12:07 AM UTC

de_Selby wrote: post #8

elhawtaky wrote: post #2

I'm the author. Let me know if you have any questions.

How does this differ from Objectivefs?

Funnily enough, i wasn't aware of ObjectiveFS - i guess it's because i can't find a research paper for it. HopsFS on S3 is similar to ADLS on Azure (built on Azure block storage). Internally, ADLS and HopsFS are different - HopsFS has a scaleout consistent metadata layer with a CDC API, while ADLS doesn't. But ADLS-v2 is also very good.