Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HopsFS: 100x Times Faster Than AWS S3

logicalclocks.com

Post reply on HN
Print view View original

91–100 of 139 posts

Re: HopsFS: 100x Times Faster Than AWS S3

#91

Post by jamesblonde »

xyzzy_plugh wrote: post #72
jamesblonde wrote: post #46

Earlier quoted context omitted.

I'm one of the co-authors. The numbers for writes are in the paper, so it is very unfair to call it an advertisement. And it is a global cache - if the block is cached somewhere, it will be used in reads.

The parent does make a good point about centralization of requests being a problem. S3 load balances under the hood, so different key prefixes within a bucket are usually serviced in isolation -- a DoS to one prefix will usually not affect other prefixes. It seems like you'd be limiting yourself for concurrent access -- if everything is flowing through the MySQL cluster -- not a bad thing! Just perhaps warrants a cav…

Read the paper, it doesn't smoke HopsFS on concurrency. In the 2nd paper, we get 1.6m ops/sec on HopsFS HA over 3 availability zones (without S3 as a backing layer). Can you get that on S3 (maybe if you are Netflix, otherwise your starting quota is 1000s ops/sec)?

Re: HopsFS: 100x Times Faster Than AWS S3

#92

Post by jamesblonde »

otterley wrote: post #33
stingraycharles wrote: post #16

Earlier quoted context omitted.

We do this in our ETL jobs on several hundreds of thousands of files a day. Not a reason to switch to a different system, but there are definitely non-maintenance use cases for this.

Any particular reason you don't tag the objects instead? That's a significantly lighter-weight operation, since S3 doesn't have native renaming capability.

In HopsFS, you can also tag objects with arbitrary metadata (the xattr API), and the JSON objects you tag with are automatically (eventually consistent) replicated to Elastic search. So, you can search the FS namespace with free-text search on elastic. However, the lag is up to 100ms from the MySQL Cluster metadata. See the epipe paper in the blog post for details.

Re: HopsFS: 100x Times Faster Than AWS S3

#93

Post by emmanueloga_ »

threeseed wrote: post #27

Looks no different to Alluxio or Minio S3 Gateway or the dozens of other S3 caches around. Would've been more interesting had it taken advantage of newer technologies such as io_uring, NVME over Fabric, RDMA etc.

I was gonna ask how it compares with Minio. Minio looks incredibly easy to "deploy" (single executable!) [1]. Looks nice to start hosting files on a single machine if there's no immediate need for the industrial strength features of AWS S3, or even to run locally for development.

1: https://min.io/download

Re: HopsFS: 100x Times Faster Than AWS S3

#94

Post by jamesblonde »

Just1689 wrote: post #14

Do you plan on having s Kubernetes storage provider? I like the idea of present a POSIX like mount to s container while paying per use for storage in S3

I like the idea, too :) We have it as a project for a Master's student at KTH starting in January.

Re: HopsFS: 100x Times Faster Than AWS S3

#95

Post by jamesblonde »

mwcampbell wrote: post #54
jamesblonde wrote: post #53

Earlier quoted context omitted.

A quick read of ObjectiveFS, and it doesn't appear to be a distributed filesystem. It appears to be a single service (log -structured storage on a service) that is backed by S3. Am I wrong? (HopsFS is a distributed hierarchical FS).

If I understand the distinction you're making, then yes, you're wrong. The really beautiful thing about ObjectiveFS is that it's distributed, but the user doesn't really have to be concerned about that. A user can mount the same S3-backed ObjectiveFS filesystem on multiple machines, and they will somehow coordinate their reads and writes to that one S3 bucket, without having to communicate with any central component…

That's not what I meant by a distributed file system. Is the metadata layer of ObjectiveFS distributed - scale-out. Can you add nodes to increase the metadata layer's capacity and throughput, so it can scale to handle millions of ops/sec? That's what I meant by a distributed file system (not just a client server, with a single metadata server).

Re: HopsFS: 100x Times Faster Than AWS S3

#97

Post by jamesblonde »

boulos wrote: post #81

Disclosure: I work on Google Cloud. Cool work! I love seeing people pushing distributed storage. IIUC though, you make a similar choice as Avere and others. You're treating the object store as a distributed block store [1]: > In HopsFS-S3, we added configuration parameters to allow users to provide their Amazon S3 bucket to be used as the block data store. Similar to HopsFS, HopsFSS3 stores the small files, 128 KB, H…

>It's somewhat unclear to me, but I think the combination of these statements means "S3 is always treated as a block store, but sometimes the File == Variably-Sized-Block == Object. Is that right?

If the file is "small" (under a configure size, typically 128KB), it is stored in the metadata-layer, not on S3. Otherwise, if you just write the file once in one session (and it is under the 5TB object size limit in S3), there will be one object in S3 (variable size - blocks in HDFS are by default fixed size). However, if you append to the file, then we add a new object (as a block) for the append.

We have a new version under development (working prototype) where we can (in the background) rewrite all the blocks in a single file as a single object, and make the object readable by a S3 API. It will be released some time next year. The idea is that you can mark directories as "S3 compatible" and only pay for rebalancing those ones as needed. But you then have the choice of doing the rebalancing on-demand or as a background task, and prioritizing, and so on. You know the tradeoffs. Yes, it would be easier to do this with GCS. But we did AWS and Azure first, as we feel GCS is more hostile to third-party vendors. The talks we have given at google (to the colossus team a couple of years ago and to Google Cloud/AI - https://www.meetup.com/SF-Big-Analytics/discussions/57666504... ) are like black holes of information transfer.

Re: HopsFS: 100x Times Faster Than AWS S3

#100

Post by redis_mlc »

ryanmarsh wrote: post #65
geertj wrote: post #44

Earlier quoted context omitted.

> It usually starts with NFS, which is a big huge single point of failure. NFS is just the protocol. Whether it's a single point of failure depends on the server-side implementation. In Amazon EFS it is not. (disclaimer: I'm a PM-T on the EFS team)

Hi, my use case for EFS Lambda is burts of small writes. Could you add a perf mode that favors bursts of small writes? We did some perf tests and just couldn’t make it fast enough.

Everything is throttled in AWS because of people like you, so the answer is no.
Post reply on HN