A Ceph war story

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.

Apart from the NTP tangent, this sounds like a Linux XFS / ServeRAID M5210 firmware issue. Your XFS filesystems created using the incorrect block/io sizes reported by the RAID controller would have been unmountable on the newer Linux kernel regardless of Ceph.

Lesson learned: your configuration management also needs to control for firmware versions such that the same issue would have shown up in a dev/test environment before turning into a prod nightmare :/

Lustre is not aligned at all with your requirements, so forget that one.

Ceph is much more complex than Gluster, but also more capable.

Honestly unless you are dealing with hundreds of TB of storage (and therefore need multiple servers anyway), I expect the complexity any distributed file-system adds is going to be detrimental to uptime and stability more often than it provides extra resilience. Use a single box with ZFS if you can, and add Gluster on top only if it can't be avoided.

Ceph and XFS actually for the longest time had an issue locking up the kernel. Ceph works fine, until it doesn't. I used gluster and Ceph for a while in production. I faintly remember Gluster supposedly being more performant for small writes, but it was an absolutely pain to deal with. Ceph is a bit better and ran stable until it didn't.

One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes. But rook has had so many issues, many of them fixed, but somehow it seems that all kubernetes "cloudnative" released software has some quirks which you just have to accept as part of the solution(this is from someone who's oftentimes pushing for kubernetes based solutions). But I also tend to recommend clients to just spend 15k to buy a TrueNAS or something similar with iSCSI and NFS.

EDIT: the XFS issue was a hung task issue that was added to the 5.6 kernel "recently". Meaning it not in Ubuntu 20.04 either.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...