Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph storage system

techbar.me

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Ceph storage system

#11

Post by komljen »

noahdesu wrote: post #9
nasalgoat wrote: post #2

Not sure if much has changed, but the last time I tried to install Ceph it wouldn't work under CentOS. That, and it was far too complicated to set up. GlusterFS, on the other hand, was incredibly easy, although I am not a fan of FUSE due to the high CPU usage.

Platform support ( http://ceph.com/docs/master/install/os-recommendations/#plat... ) and deployment tools have come a long way. In the past setup has been complicated (Ceph is inherently more complex than other systems), but it is getting much easier. There is also extensive documentation at http//ceph.com/docs, as well as very active IRC channel and mailing lists for support. Ceph is much larger than just the file s…

Correct, Ceph is much larger than the file system. As I said at beginning of article I'm using each Ceph component, RBD, RadosGW and CephFS and I will write article for each of that. This is just some sort of getting started guide. I'm interested at running Hadoop on top of CephFS, is it stable enough?

Re: Ceph storage system

#12

Post by fintler »

nasalgoat wrote: post #2

Not sure if much has changed, but the last time I tried to install Ceph it wouldn't work under CentOS. That, and it was far too complicated to set up. GlusterFS, on the other hand, was incredibly easy, although I am not a fan of FUSE due to the high CPU usage.

I still see Ceph as competition to supercomputer filesystems and not really as competition to GlusterFS. For example, the design of it directly attacks the problem of centralized metadata (especially useful after the DARPA project to bring it to Lustre failed).

I was working on an unrelated project with one of the designers of Ceph (UCSC's Scott Brandt) and in conversation he also seems to concur that Ceph was really built as a replacement for PanFS or Lustre (but still may be useful for other things of course).

Using it to replace GlusterFS still seems odd to me. It feels like they're both solutions to different problems.

Re: Ceph storage system

#13

Post by noahdesu »

komljen wrote: post #11
noahdesu wrote: post #9

Earlier quoted context omitted.

Platform support ( http://ceph.com/docs/master/install/os-recommendations/#plat... ) and deployment tools have come a long way. In the past setup has been complicated (Ceph is inherently more complex than other systems), but it is getting much easier. There is also extensive documentation at http//ceph.com/docs, as well as very active IRC channel and mailing lists for support. Ceph is much larger than just the file s…

Correct, Ceph is much larger than the file system. As I said at beginning of article I'm using each Ceph component, RBD, RadosGW and CephFS and I will write article for each of that. This is just some sort of getting started guide. I'm interested at running Hadoop on top of CephFS, is it stable enough?

The instability we have seen is with the metadata server, but have been able to relatively quickly push fixes to upstream as we encounter them. The focus has been on stability, and we have been running pretty large terasort jobs without issue. In the upcoming release of Cuttlefish, due out any time now, there will be locality information exposed to Hadoop for better task scheduling. We will start focusing on improving Hadoop performance soon, now that things have stabilized.

Re: Ceph storage system

#16

Post by noahdesu »

fintler wrote: post #12
nasalgoat wrote: post #2

Not sure if much has changed, but the last time I tried to install Ceph it wouldn't work under CentOS. That, and it was far too complicated to set up. GlusterFS, on the other hand, was incredibly easy, although I am not a fan of FUSE due to the high CPU usage.

I still see Ceph as competition to supercomputer filesystems and not really as competition to GlusterFS. For example, the design of it directly attacks the problem of centralized metadata (especially useful after the DARPA project to bring it to Lustre failed). I was working on an unrelated project with one of the designers of Ceph (UCSC's Scott Brandt) and in conversation he also seems to concur that Ceph was really…

Recent gluster vs ceph debate at LCA.

http://www.youtube.com/watch?v=JfRqpdgoiRQ

Re: Ceph storage system

#17

Post by jmspring »

noahdesu wrote: post #15
mad44 wrote: post #14

Here is a summary of how Ceph works. http://muratbuffalo.blogspot.com/2011/03/ceph-scalable-high-...

Inktank is the professional services company backing Ceph. Here is their youtube channel with a whole lot of up-to-date videos: http://www.youtube.com/user/inktankstorage

Glad to see Sage commercializing this and happy to see research activities from UC Santa Cruz's (where I went for grad/undergrad) Storage Systems Research Center making it into commercial use.

More SSRC work and some of the original CEPH papers can be found at:

http://www.ssrc.ucsc.edu/index.html

Re: Ceph storage system

#18

Post by fintler »

noahdesu wrote: post #16
fintler wrote: post #12

Earlier quoted context omitted.

I still see Ceph as competition to supercomputer filesystems and not really as competition to GlusterFS. For example, the design of it directly attacks the problem of centralized metadata (especially useful after the DARPA project to bring it to Lustre failed). I was working on an unrelated project with one of the designers of Ceph (UCSC's Scott Brandt) and in conversation he also seems to concur that Ceph was really…

Recent gluster vs ceph debate at LCA. http://www.youtube.com/watch?v=JfRqpdgoiRQ

Nice talk, but I think I was basically trying to say that Ceph really shines when you compare it to Lustre and PanFS.

It's not a clear winner when you compare it to GlusterFS because the original design wasn't intended to replace GlusterFS (although it may do a good job of this anyway).

Re: Ceph storage system

#19

Post by stonith »

fintler wrote: post #18
noahdesu wrote: post #16

Earlier quoted context omitted.

Recent gluster vs ceph debate at LCA. http://www.youtube.com/watch?v=JfRqpdgoiRQ

Nice talk, but I think I was basically trying to say that Ceph really shines when you compare it to Lustre and PanFS. It's not a clear winner when you compare it to GlusterFS because the original design wasn't intended to replace GlusterFS (although it may do a good job of this anyway).

Ceph is still a very long way behind Lustre for streaming bandwidth, so to say it shines would be a little much. Lustre's weakness is in scaling to support large file counts, but in real deployments this can be mitigated by using an MDS server with a lot of grunt. Ceph can't compete with Lustre for HPC deployments until it supports RDMA, and even with that it's still going to take a long time to reach Lustre's performance (which is close to line rate at this point)
Post reply on HN