Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HopsFS: 100x Times Faster Than AWS S3

logicalclocks.com

Post reply on HN
Print view View original

51–60 of 139 posts

Re: HopsFS: 100x Times Faster Than AWS S3

#51

Post by jamesblonde »

chordysson wrote: post #40
therealmarv wrote: post #18

Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?

Regarding small files, HopsFS can store small files in the metadata layer for improved performance. https://kth.diva-portal.org/smash/record.jsf?pid=diva2:12608...

Yes, this paper did not compare small file performance in HopsFS with EMRFS/S3. But HopsFS on S3 also supports storing small files on NVMe disks in the metadata layer. Previously, we have shown that we can get up to 66X higher throughput for writes for small files versus HDFS (peer reviewed, of course) - https://www.logicalclocks.com/blog/millions-and-millions-of-...

Re: HopsFS: 100x Times Faster Than AWS S3

#52

Post by Cixelyn »

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

> EdgeFS is one I was looking at recently

Do you have any additional info? EdgeFS's github[1] doesn't work; does repo access require a Nexenta sales call?

We're also looking into asynchronously replicated FSs, I think built-in caching + tiering is slightly nicer than cron + rsync; would love to know what other solutions you looked into.

[1] https://github.com/Nexenta/edgefs

Re: HopsFS: 100x Times Faster Than AWS S3

#53

Post by jamesblonde »

mwcampbell wrote: post #29

> But, until today, there has been no equivalent to ADLS for S3. ObjectiveFS has been around for several years. How is HopsFS better?

A quick read of ObjectiveFS, and it doesn't appear to be a distributed filesystem. It appears to be a single service (log -structured storage on a service) that is backed by S3. Am I wrong? (HopsFS is a distributed hierarchical FS).

Re: HopsFS: 100x Times Faster Than AWS S3

#54

Post by mwcampbell »

jamesblonde wrote: post #53
mwcampbell wrote: post #29

> But, until today, there has been no equivalent to ADLS for S3. ObjectiveFS has been around for several years. How is HopsFS better?

A quick read of ObjectiveFS, and it doesn't appear to be a distributed filesystem. It appears to be a single service (log -structured storage on a service) that is backed by S3. Am I wrong? (HopsFS is a distributed hierarchical FS).

If I understand the distinction you're making, then yes, you're wrong. The really beautiful thing about ObjectiveFS is that it's distributed, but the user doesn't really have to be concerned about that. A user can mount the same S3-backed ObjectiveFS filesystem on multiple machines, and they will somehow coordinate their reads and writes to that one S3 bucket, without having to communicate with any central component besides S3 itself and the ObjectiveFS licensing server.

Re: HopsFS: 100x Times Faster Than AWS S3

#55

Post by jamesblonde »

Lucasoato wrote: post #5

Can this be installed on AWS EMR clusters?

Yeah, https://hopsworks.ai is an alternative to EMR that includes HopsFS out-of-the-box. You get $4k free credits right now - it includes Spark, Flink, Python, Hive, Feature Store, GPUs (tensorflow), Jupyter, notebooks as jobs, AirFlow. And a UI for development and operations. So kind of like Databricks with more of a ML focus, but still supporting Spark.

Re: HopsFS: 100x Times Faster Than AWS S3

#56

Post by lowdose »

oneplane wrote: post #23
rgbrenner wrote: post #20

Earlier quoted context omitted.

Ok so you're not the target customer for this product. Do you believe people with this problem should be deprived of a solution just because you don't need it?

I don't think I suggest that a product should not exist. I suggest that targeting the competition instead of the use case is a bit silly.

Conflict generates more attention.

Re: HopsFS: 100x Times Faster Than AWS S3

#57

Post by hanikesn »

tutfbhuf wrote: post #32

High-availability durable filesystem is a difficult problem to solve. It usually starts with NFS, which is a big huge single point of failure. Depending on the nature of the application this might be good enough. But if it's not, you'll typically want cross-datacenter replication so if one rack goes down you don't lose all your data. So then you're looking at something like Glusterfs/MooseFS/Ceph. But the latencies i…

Quobyte claim to have solved the synchronized distributed POSIX filesystem (mod geographically ).

Re: HopsFS: 100x Times Faster Than AWS S3

#59

Post by colinwilson »

hartator wrote: post #11

> 100X the performance of S3 for file move/rename operations I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than something you want to do on a regular basis.

Systems like Hadoop and Spark will run multiple versions of the same task writing out to a FS at once but to a temporary directory and when the first finishes the output data is just moved to the final place. It's not uncommon for a job to "complete" writing data to S3 and just sit there and hang as the FS move commands run copying the data and deleting the old version. It is just assumed a rename/move is a no-op in some systems.

Re: HopsFS: 100x Times Faster Than AWS S3

#60

Post by spicyramen »

I have VM for my data scientists already in GCP, my datasets live in Google Cloud Storage. Can I take advantage of HopsFS for a shared file system across my VMs. Google Filestore Is ridículous expensive and at least they give u 1TB. Multi writer only supports 2VMs
Post reply on HN