Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BeeGFS, the Parallel Cluster File System

beegfs.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: BeeGFS, the Parallel Cluster File System

#31

Post by hatsunearu »

angrygoat wrote: post #25
hatsunearu wrote: post #22

Earlier quoted context omitted.

http://www.ceph.com/ pretty hard to install (ceph-deploy makes it easier) but I've seen some pretty spiffy claims like really easy scaling and provisioning.

I've run a ceph deployment for about 18 months now, it's good. ops-wise it's like anything, a few quirks to learn - but it's very solid. I played around with a couple of puppet modules and ended up on ceph-deploy as well, makes life quite easy.

Out of curiosity, how big is your cluster in terms of nodes, capacity, throughput, etc?

Re: BeeGFS, the Parallel Cluster File System

#33

Post by hkothari »

Can someone explain how this differs from something like HDFS? I'm kind of confused as to why there's no comparisons or mentions of HDFS here which makes me think I'm missing something important about what this provides that's special. Is it the easy to install part?

Re: BeeGFS, the Parallel Cluster File System

#34

Post by scurvy »

mattbillenstein wrote: post #17
eis wrote: post #11

Earlier quoted context omitted.

I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance. Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe to…

I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers. Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately. Moosefs is pragmat…

Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.

Re: BeeGFS, the Parallel Cluster File System

#35

Post by scurvy »

mattbillenstein wrote: post #17
eis wrote: post #11

Earlier quoted context omitted.

I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance. Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe to…

I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers. Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately. Moosefs is pragmat…

Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.

Re: BeeGFS, the Parallel Cluster File System

#36

Post by scurvy »

eis wrote: post #11
pjc50 wrote: post #9

Earlier quoted context omitted.

Could you elaborate on this please?

I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance. Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe to…

Ceph block storage or CephFS?

Re: BeeGFS, the Parallel Cluster File System

#37

Post by birc_a »

poelzi wrote: post #6

found this interesting: https://indico.cern.ch/event/346931/session/4/contribution/6... > Plain BeeGFS is as reliable as your hardware currently i would give MooseFS, orangefs or lizardfs a try. Had experience with ceph and glusterfs and both I can not recommend.

MooseFS and BeeGFS don't really seem to be in the same category of filesystem though.

BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going.

BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth.

Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable.

If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though.

We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.

Re: BeeGFS, the Parallel Cluster File System

#38

Post by illumin8 »

sharktour wrote: post #15

This doesn't fill the huge gap for Cloud NAS. GlusterFS is hugely complicated and a fragile install, SoftNAS has (anecdotally) poor performance on high-io... but it's closer. Amazon hasn't launched their EFS product to General Availability Yet. So wtf do we do when we need a 5TB shared storage platform across some EC2 nodes?

Check out Avere systems. They have a clustered NFS appliance that you can spin up in EC2, which translates back-end S3 object storage into NFS for your clients. It supports between 3 and 50 nodes, and uses local RAM and SSD cache to accelerate performance. Avere has also been in the NAS acceleration/caching business since 2008, so their product is very mature.

Re: BeeGFS, the Parallel Cluster File System

#39

Post by birc_a »

hkothari wrote: post #33

Can someone explain how this differs from something like HDFS? I'm kind of confused as to why there's no comparisons or mentions of HDFS here which makes me think I'm missing something important about what this provides that's special. Is it the easy to install part?

HDFS is designed mostly for immutable files, stores three copies of everything in the default setup and has the whole hadoop calculation stuff.

BeeGFS is closer to a normal NFS share. In the default setup it just stores the data once, striped over multiple machines and it supports efficient updates in-place etc.

Re: BeeGFS, the Parallel Cluster File System

#40

Post by 6502nerdface »

rbanffy wrote: post #12
leni536 wrote: post #5

http://www.beegfs.com/docs/BeeGFS_EULA.txt Umm, how is this compatible with GPL v2? It seems more restrictive (enterprise features, parts that would be better enforced by a trademark policy).

Speaking of trademarks, there is a band who would want to talk to these guys.

How is there any potential for commercial confusion?
Post reply on HN