A Ceph war story • Hacker News Board

Re: A Ceph war story

#51

Post by jlengrand » Sat, Apr 10, 2021, 8:19 PM UTC

ali_m wrote: post #49

jlengrand wrote: post #42

Completely tangential but I had never heard of Ceph. Went to the website [1], literally no info as to what it does. However I can download, read tweets, or do some training. It's a shame because the documentation page [2] has an amazing, concise and precise description of what it's doing : Ceph uniquely delivers object, block, and file storage in one unified system. It takes one line, why is it not there? [1] https:/…

To be fair, it does say: > Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. although it's annoyingly hidden away in a carousel. Carousels - not even once.

Oh wow indeed. In third place out of 4

Re: A Ceph war story

#52

Post by axiolite » Sat, Apr 10, 2021, 9:33 PM UTC

m463 wrote: post #40

bsder wrote: post #11

Earlier quoted context omitted.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

I got a hardware raid controller at home, many years ago. And I had a failure. I couldn't get to anything without buying an identical controller. Swore I'd never do again.

Linux md-raid supports LSI MegaRAID drives just fine.

https://askubuntu.com/questions/1310586/megaraid-lsi-raid-to...

Re: A Ceph war story

#53

Post by lillecarl » Sun, Apr 11, 2021, 4:22 AM UTC

bsder wrote: post #16

fulafel wrote: post #12

Earlier quoted context omitted.

They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size. So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet. Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?

Ah, good catch, I somehow missed that they had it in "JBOD" mode. As you point out, though, it doesn't seem like it's "really" JBOD mode. It seems like that card is bonding the disks together somehow into a larger logical "stripe". Weird.

JBOD can also mean "raid 0 with different sized disks". It's a great way to lose all your data!

Re: A Ceph war story

#54

Post by Tepix » Mon, Apr 12, 2021, 8:14 AM UTC

If i have 100 clients (laptops) with 100GB available on each laptop's disk, is Ceph a suitable method of making this space available as a large network volume?

Edit: I realize that it's only 10TB and probably not worth the hassle.

Re: A Ceph war story

#55

Post by ibotty » Tue, Apr 13, 2021, 7:23 AM UTC

mikagrml wrote: post #26

terom wrote: post #6

Apart from the NTP tangent, this sounds like a Linux XFS / ServeRAID M5210 firmware issue. Your XFS filesystems created using the incorrect block/io sizes reported by the RAID controller would have been unmountable on the newer Linux kernel regardless of Ceph. Lesson learned: your configuration management also needs to control for firmware versions such that the same issue would have shown up in a dev/test environmen…

(Author here) Yes, it was an XFS/controller issue, but Ceph reported the failure. :) (IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database, but nowadays ceph-disk (which uses those XFS partitions) is gone, and instead ceph-volume uses a different approach via LVM.) Regarding configuration management/firmware version: yes - especially, as you'd need to also rebuild disks in…

> IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database

It has been shown, that you are right. But not because of bugs like the one you encountered. The problem could as well have happened with a regular xfs fs holding a maildir.

Re: A Ceph war story

#56

Post by ibotty » Tue, Apr 13, 2021, 7:34 AM UTC

Tepix wrote: post #54

If i have 100 clients (laptops) with 100GB available on each laptop's disk, is Ceph a suitable method of making this space available as a large network volume? Edit: I realize that it's only 10TB and probably not worth the hassle.

No! Not at all. You have to control uptime of the single nodes holistically. That just does not work with Ceph. I can't imagine that there are storage systems that work with that setup.