> 100X the performance of S3 for file move/rename operations
I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than something you want to do on a regular basis.
HopsFS: 100x Times Faster Than AWS S3
11–20 of 139 posts
Re: HopsFS: 100x Times Faster Than AWS S3
#12Diagram seems to imply an active/passive namenode setup like HDFS. Doesn't that limit it to tiny filesystems, and curse it with the same availability problems that plague HDFS?
Re: HopsFS: 100x Times Faster Than AWS S3
#13Does it matter? I often see 'our product is much faster than thing X at cloud Y!' and find myself asking why. Why would I want something less integrated for a performance change I don't need, a software change I'll have to write and an extra overhead for dealing with another source?
It's great that one individual thing is better than one other individual thing, but if you look at the bigger picture it generally isn't that individual thing by itself that you are using.
Re: HopsFS: 100x Times Faster Than AWS S3
#14Do you plan on having s Kubernetes storage provider? I like the idea of present a POSIX like mount to s container while paying per use for storage in S3
Re: HopsFS: 100x Times Faster Than AWS S3
#15I'm the author. Let me know if you have any questions.
> ... but has 100X the performance of S3 for file move/rename operations Isn't rename in S3 effectively a copy-delete operation?
That’s my understanding too. Also rename / copy turned out not to be very useful at the end of the day. Nearly all my implementations just boil down to randomized characters as ids
Re: HopsFS: 100x Times Faster Than AWS S3
#16> 100X the performance of S3 for file move/rename operations I don’t see how it can be useful. Moving or renaming files in S3 seems more like maintenance than something you want to do on a regular basis.
We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenance use cases for this.
Re: HopsFS: 100x Times Faster Than AWS S3
#17What’s the latency with 100,000 and 1,000,000 files?
We haven't run mv/list experiments on 100,000 and 1,000,000 files for the blog. However, We expect the gap in latency between HopsFS/S3 and EMRFS would increase even further with larger directories. In our original HopsFS paper, we showed that the latency for mv/rename for a directory with 1 million files was around 5.8 seconds.
[deleted]
Re: HopsFS: 100x Times Faster Than AWS S3
#18Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file.
Is HopsFS helping in this area?
Re: HopsFS: 100x Times Faster Than AWS S3
#19Does it matter? I often see 'our product is much faster than thing X at cloud Y!' and find myself asking why. Why would I want something less integrated for a performance change I don't need, a software change I'll have to write and an extra overhead for dealing with another source? It's great that one individual thing is better than one other individual thing, but if you look at the bigger picture it generally isn't…
It does matter, because not every use case requires everything but the kitchen sink. If you're building things that only ever require the same mass produced components for every application, well that stifles the possibilities of what can be built.
Re: HopsFS: 100x Times Faster Than AWS S3
#20Does it matter? I often see 'our product is much faster than thing X at cloud Y!' and find myself asking why. Why would I want something less integrated for a performance change I don't need, a software change I'll have to write and an extra overhead for dealing with another source? It's great that one individual thing is better than one other individual thing, but if you look at the bigger picture it generally isn't…
Ok so you're not the target customer for this product. Do you believe people with this problem should be deprived of a solution just because you don't need it?