A Ceph war story • Hacker News Board

Re: A Ceph war story

#11

Post by bsder » Sat, Apr 10, 2021, 9:05 AM UTC

fulafel wrote: post #2

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

> fancy RAID controller

I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster.

Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs.

The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to put it in the trash and get something that does. If Ceph doesn't really work, RAID hardware is just adding an extra failure point that adds nothing.

Re: A Ceph war story

#12

Post by fulafel » Sat, Apr 10, 2021, 9:19 AM UTC

bsder wrote: post #11

fulafel wrote: post #2

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size.

So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet.

Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?

Re: A Ceph war story

#13

Post by ibotty » Sat, Apr 10, 2021, 9:19 AM UTC

trabant00 wrote: post #5

Anybody got experience comparing ceph vs gluster vs lustre, etc? I am interested in simplicity and resilience in case of node outage. Is any of these close to set and forget?

I have mildly bad experience with gluster. I did never lose data, but the maintenance load was very noticable! Ceph will rebalance by itself when adding (or removing) disks. That alone reduces maintenance by a huge amount. For Storage clusters I don't care about as much, it's also nice when a disk fails that ceph will notice and react automatically. Replacing the disk is then something you ought to to at one time, but you don't have to do pretty soon.

I am running some rook-managed ceph clusters now that make even ceph infrastructure nodes automatically recoverable. But I did not test that enough yet.

Re: A Ceph war story

#14

Post by ps » Sat, Apr 10, 2021, 9:27 AM UTC

rjzzleep wrote: post #10

Ceph and XFS actually for the longest time had an issue locking up the kernel. Ceph works fine, until it doesn't. I used gluster and Ceph for a while in production. I faintly remember Gluster supposedly being more performant for small writes, but it was an absolutely pain to deal with. Ceph is a bit better and ran stable until it didn't. One of the issues is that the officially recommended way to install it seems to…

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k...

“One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes.“

I believe the recommended way is using cephadm.

I have been using ceph-ansible since luminous and had to deal with nasty issues or quirks as well caused probably by complexity and human errors.

Re: A Ceph war story

#15

Post by emmericp » Sat, Apr 10, 2021, 9:47 AM UTC

Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one.

Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode.

Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

ceph-volume finally fully removed dependency on file systems. Yeah the LVM-mess is sometimes annoying and early version of ceph-volume had many problems, but nowadays I wouldn't want ceph-disk back.

Re: A Ceph war story

#16

Post by bsder » Sat, Apr 10, 2021, 9:50 AM UTC

fulafel wrote: post #12

bsder wrote: post #11

Earlier quoted context omitted.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size. So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet. Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?

Ah, good catch, I somehow missed that they had it in "JBOD" mode.

As you point out, though, it doesn't seem like it's "really" JBOD mode. It seems like that card is bonding the disks together somehow into a larger logical "stripe". Weird.

Re: A Ceph war story

#17

Post by merb » Sat, Apr 10, 2021, 10:11 AM UTC

emmericp wrote: post #15

Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…

> Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode.

well raid 1 works aswell, if you can take the performance hit. most raid 1 drives inside a raid controller can be run outside of the raid. we already needed to do that since one of our customers tought it is a good idea to have a running server temporarly near an open window with production data and no backup (we exclude our liability in this case and monitor if backups are created) so we needed to recover the data. which worked by using another controller with passtrough and just running a single disk. (the other disk was destroyed, so was the raid controller) btw. rainwater damages a server, especially if you do not notice it for 30 minutes and a full bucket of water inside it. kudos to dell the server kept running for 25 minutes, when it was full of water until it died. (we transported the server and still had water in it...)

Re: A Ceph war story

#18

Post by sekh60 » Sat, Apr 10, 2021, 10:31 AM UTC

ps wrote: post #14

rjzzleep wrote: post #10

Ceph and XFS actually for the longest time had an issue locking up the kernel. Ceph works fine, until it doesn't. I used gluster and Ceph for a while in production. I faintly remember Gluster supposedly being more performant for small writes, but it was an absolutely pain to deal with. Ceph is a bit better and ran stable until it didn't. One of the issues is that the officially recommended way to install it seems to…

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k... “One of the issues is that the officially recommended way to install it seems to be rook-ceph wi…

You are correct about cephadm being the recommended install method.

For NVMe OSDs how many daemons are using the same drive? Normally to get performance you have to colocate two or four osd daemons on a single NVMe device.

Also ceph benefits the most from massive parallelism, how many clients do you have? 4 OSDd is not a lot to spread the load over, even if the drives are lightning fast.

There is a new io pathway in the works called crimson which should make NVMe drives more performant. I was hoping a preview would lend in Pacific, but I guess it isn't close enough to being ready.

Re: A Ceph war story

#19

Post by elongatedMusku » Sat, Apr 10, 2021, 10:33 AM UTC

luto wrote: post #9

hutrdvnj wrote: post #4

Earlier quoted context omitted.

Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.

can confirm this on two 24 node / 100TB / 132 OSD clusters and ~2 years.

Getting my feet wet with 3 nodes, 3 osds. Wish me luck.

Re: A Ceph war story

#20

Post by sekh60 » Sat, Apr 10, 2021, 10:34 AM UTC

Tepix wrote: post #3

Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?

I run a home ceph cluster as a hobby. It works really well. I have hit a few snags, but the mailing list is super friendly and helpful and the devs are quick to respond to bug reports.