I will say that I know two people that have independently tried fairly large trials (400TB+) of Lustre, GlusterFS, and BeeGFS, and BeeGFS was their eventual favorite.
Nice anecdote, but can you provide at least ANY detail on why they picked BeeGFS?
BeeGFS, the Parallel Cluster File System
41–47 of 47 posts
Re: BeeGFS, the Parallel Cluster File System
#42It's a tragedy to think of how deep is your love for the project.
Wonder how long it will be stayin' alive? Gives me a night fever just thinking about it.
Re: BeeGFS, the Parallel Cluster File System
#43The company I work for has been running BeeGFS for 6 years with no issues. Write and read speeds on Dell storage hardware match or exceed the NetApp appliances we use.
We do occasionally have issues where ALL storage nodes must be rebooted but other than it works great.
Re: BeeGFS, the Parallel Cluster File System
#44Re: BeeGFS, the Parallel Cluster File System
#45Earlier quoted context omitted.
I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers. Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately. Moosefs is pragmat…
Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.
Re: BeeGFS, the Parallel Cluster File System
#46Earlier quoted context omitted.
Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.
cephfs is clearly marked as not for production. It still has more than its fair share of bugs and has single points of failure. From the very top of the docs page on ceph FS:
"Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters"
RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.
Re: BeeGFS, the Parallel Cluster File System
#47This doesn't fill the huge gap for Cloud NAS. GlusterFS is hugely complicated and a fragile install, SoftNAS has (anecdotally) poor performance on high-io... but it's closer. Amazon hasn't launched their EFS product to General Availability Yet. So wtf do we do when we need a 5TB shared storage platform across some EC2 nodes?
Check out Avere systems. They have a clustered NFS appliance that you can spin up in EC2, which translates back-end S3 object storage into NFS for your clients. It supports between 3 and 50 nodes, and uses local RAM and SSD cache to accelerate performance. Avere has also been in the NAS acceleration/caching business since 2008, so their product is very mature.
Thanks for the heads up on Avere. Use case for us was for file parsing as part of a batch data pipeline. The latency we saw on alternatives was killer compared to directly mounted EBS. That would be my fear with an S3 option as well, especially when it presents like NFS.