A Ceph war story

And even if you need hundreds of TB of storage, in many cases it might be better to just shard over normal storage and use your backups in case of failures. Current disks are big so you don't need too many for eg 200 TB.

Gluster is also so much easier to set up IMO; the only reason I wouldn't use it is that it seems to have some consistency issues such that ex. you could run postgres on ceph, but not (safely) on gluster. But I don't understand it well enough to know why.

The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with HBA/simple-RAID firmware ("IT" or "IR") to fix this.

But really, this had little to do with Ceph (other than the ops mistake of rebooting several machines at once without waiting for an all clear, which is just a bad idea all around when you're upgrading a cluster of any kind). It was an interaction between XFS (not used on modern ceph-volume systems), the kernel, and the RAID controller.

I find Ceph to actually be a lot more introspectable than "typical" multi disk storage management filesystems like btrfs and zfs. On those, if something goes wrong, you're often left with a corrupted or unmountable filesystem, or worse, kernel panics or errors. It's all a monolith in the kernel and hard to fix anything. On Ceph, you can dig through the layers, and things are split into separate daemons that can be debugged separately.

For example, want to see where the data of a CephFS file is stored? Easy: the inode number is the first part of the object name in the data pool. Append the block number after that and you have your object name. You can just fetch that using the rados tool, bypassing CephFS entirely. Want to go deeper? Ask it to hash the object name to a PG ID. Use `ceph pg dump` to figure out what OSDs that pg lives in. Now you know what disks have your data. OSDs broken? Use the objectstore tool to read the data directly without booting an OSD.

Obviously you need to learn all this stuff, but the tooling is there and this is way more introspectable than a bunch of magic structures in a filesystem.

A few things jump out to me after skimming the article:

  1. It appears that Ceph/XFS is pretty complex and complicated, and that it doesn't have the best discoverability.
  2. Distributed storage in general feels like it could cause a lot of problems unless you have someone who has lots of knowledge about it.

The dashboard seems pretty handy to get an overview of the overall state of the cluster, but the error messages in the logs do feel really low level and not that actionable on their own (unless you are really familiar with XFS and Ceph). Without such familiarity, i'd probably be stuck for a long time.

On an unrelated note, the command at the end of the article would cause me problems because i wouldn't tell what it does at a glance (though maybe that's just because i'd prefer longer scripts instead of long one liners). Is there a reason why people don't store comments with every command, so that they remember what it does after a few months?

For example:

  # Check whether XFS mount points are affected by a problem in the 4.18-rc1 kernel version: https://bugzilla.kernel.org/show_bug.cgi?id=202127
  # Will feed the current mounts into xfs_info to figure out if sunit and swidth values are incorrect.
  awk '$3 == "xfs"{print $2}' /proc/self/mounts | while read mount ; do echo -n "$mount " ; xfs_info $mount | awk '$0 ~ "swidth"{gsub(/.*=/,"",$2); gsub(/.*=/,"",$3); print $2,$3}' | awk '{ if ($1 > $2) print "impacted"; else print "OK"}' ; done

Or do you just get used to "decoding" what the awk and gsub invocations do eventually?

Also, i bet that Ceph is good in an enterprise setting where you really need to store a lot of data, but what about smaller and simpler distributed file systems? I feel like maybe something is missing between just having some servers with local storage or maybe NFS, and full blown solutions like Ceph. I have heard about GlusterFS, LizardFS, MooseFS, SeaweedFS and others, yet all of them seem noticeably more complicated than setting up a Docker Swarm cluster would be (essentially just telling a bunch of nodes to communicate amongst themselves with 1 CLI command per node, and letting them sort the rest out themselves).

Plus, i've heard that even some file systems can have a noticeable impact on the resource consumption on the server (though i just searched and can't find anything concrete in DuckDuckGo, even the Wikipedia page i used doesn't seem to have any good recommendations for CPU/RAM resources: https://en.wikipedia.org/wiki/Comparison_of_distributed_file... ), so setting clusters like that up doesn't feel like something that a person would do in their small homelab with a few old Athlon processors. :(

Of course, it may be that distributed file storage is just inherently more complex and demanding than making a few servers talk amongst themselves and achieve consensus (the Docker Swarm example above). Any suggestions/opinions?

To be fair, it does say:

> Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.

although it's annoyingly hidden away in a carousel. Carousels - not even once.