I'm the author. Let me know if you have any questions.
You say it's "posix-like" - so what from posix had to be left out?
HopsFS: 100x Times Faster Than AWS S3
21–30 of 139 posts
Re: HopsFS: 100x Times Faster Than AWS S3
#22Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?
How small are the files?
Here are some strategies to make S3 go faster: https://news.ycombinator.com/item?id=19475726
--
You're right that S3 isn't for small files but for a lot of small files (think 500 bytes), I either plunge them to S3 through Kinesis Firehose or fit them gzipped into DynamoDB (depending on access patterns).
One could also consider using Amazon FSx which can IO to S3 natively.
Re: HopsFS: 100x Times Faster Than AWS S3
#23Does it matter? I often see 'our product is much faster than thing X at cloud Y!' and find myself asking why. Why would I want something less integrated for a performance change I don't need, a software change I'll have to write and an extra overhead for dealing with another source? It's great that one individual thing is better than one other individual thing, but if you look at the bigger picture it generally isn't…
Ok so you're not the target customer for this product. Do you believe people with this problem should be deprived of a solution just because you don't need it?
I don't think I suggest that a product should not exist. I suggest that targeting the competition instead of the use case is a bit silly.
Re: HopsFS: 100x Times Faster Than AWS S3
#24Reading/writing SMALL files are SUPER slow on things like S3, Google Drive, Backblaze. Also using a lot of threads does only help a little bit but it's nowhere near reading/writing speeds of e.g. a single 600MB file. Is HopsFS helping in this area?
It's not surprising that it's slow. Handling files has 2 costs associated to it: One is a fixed setup cost, which includes looking up the file metadata, creating connections between all the services that make it possible to access the file, and starting the read/write. The other part is actually transferring the file content.
For small files the fixed cost will be the most important factor. The "transfer time" after this "first byte latency" might actually be 0, since all the data could be transferred within a single write call on each node.
Re: HopsFS: 100x Times Faster Than AWS S3
#25Does it matter? I often see 'our product is much faster than thing X at cloud Y!' and find myself asking why. Why would I want something less integrated for a performance change I don't need, a software change I'll have to write and an extra overhead for dealing with another source? It's great that one individual thing is better than one other individual thing, but if you look at the bigger picture it generally isn't…
It does matter, because not every use case requires everything but the kitchen sink. If you're building things that only ever require the same mass produced components for every application, well that stifles the possibilities of what can be built.
So say you have a solution for your object storage and it is plenty. Then a different product pops up, solves the same problem in exactly the same way, and costs the same. Migrating isn't free and at the end you have two suppliers to manage. Does that make any sense at all? I think not.
There is a different case that makes sense: you have object storage and it's not sufficient, so you go look for object storage suppliers that deliver something different so it suits your need. Now it makes sense to look for a service that is relatively similar to what you are already using but is better in a factor that is significant for your application (i.e. speed of object key changes), now it does make sense.
Marketing just says: "Look at us, we are faster". I think that message is not going to matter unless that happens to be your exact problem in isolation, which isn't exactly common; systems don't tend to run in isolation.
Re: HopsFS: 100x Times Faster Than AWS S3
#26I'm the author. Let me know if you have any questions.
Reading through your article, this solution is built on top of s3. So, moving and listing files is faster, presumably due to a new metadata system you've built for tracking files. The trade off here, is that writes must be strictly slower now than they were previously because you've added a network hop. All read and write data now flows through these workers. Which adds a point of failure, if you steam too much data through these workers, you could potentially OOM them. Reads are potentially faster, but that also depends on the hit rate of the block cache. Would be nice to see a more transparent post listing the pros and cons, rather than what reads as a technical advertisement.
Re: HopsFS: 100x Times Faster Than AWS S3
#27Looks no different to Alluxio or Minio S3 Gateway or the dozens of other S3 caches around.
Would've been more interesting had it taken advantage of newer technologies such as io_uring, NVME over Fabric, RDMA etc.
Re: HopsFS: 100x Times Faster Than AWS S3
#28Do you plan on having s Kubernetes storage provider? I like the idea of present a POSIX like mount to s container while paying per use for storage in S3
You can do this using s3backer, goofys etc to mount S3 as a host filesystem and then use the host path provisioner.
Re: HopsFS: 100x Times Faster Than AWS S3
#29> But, until today, there has been no equivalent to ADLS for S3.
ObjectiveFS has been around for several years. How is HopsFS better?
Re: HopsFS: 100x Times Faster Than AWS S3
#30Can this be installed on AWS EMR clusters?
You could but I would do thorough real-world benchmarks.
EMRFS has dozens of optimisations for Spark/Hadoop workloads e.g. S3 select, partitioning pruning, optimised committers etc and since EMR is a core product it is continually being improved. Using HopsFS would negate all of that.