Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?
Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.
A Ceph war story
31–40 of 56 posts
Re: A Ceph war story
#32Re: A Ceph war story
#33Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…
I've only used Ceph as provided to be by others and considered setting it up in some instances. Didn't know about the development of BlusStore and it does seem much simpler. The choice between xfs, btrfs, ext4 always seemed a bit unclear (except that I had experienced non-Ceph troubles with btrfs).
Note to self: use ceph-volume/BlueStore.
Re: A Ceph war story
#34Anybody got experience comparing ceph vs gluster vs lustre, etc? I am interested in simplicity and resilience in case of node outage. Is any of these close to set and forget?
Ceph and Lustre are both largely "set and forget" for the object-storage nodes which are the most numerous type. On the other hand, both have separate metadata servers which can be much more troublesome if/when one fails. Gluster only has one kind of node. One of the main issues here is: when do you actually give up on a node and start regenerating its data somewhere else? Starting that process and then having the node actually come back can be anywhere from a waste of time to a total disaster, so you don't want to be too "twitchy" about it, but you also don't want to run in a degraded state forever. In Gluster at least, the approach was to assume that a node's coming back unless/until explicitly told otherwise (possibly by external automation).
The larger point IMO is that no distributed storage system is "set and forget" at any significant scale. At the very least you'll want robust monitoring (which to some extent has to be built into the code) and alerting, and somebody to respond to the more serious alerts. I'd give a nod to Ceph in this respect. Even better is to have your own site-specific automation for common tasks like capacity addition and upgrades. Even if the storage system itself is doing everything "right" it can get pretty messy if that's not happening in sync with other systems such as provisioning and service discovery - not to mention the systems actually using the storage.
Also, even though the "POSIX is dead" folks are my sworn enemies, it's still true that an ever-decreasing number of workloads actually require an actual filesystem and its associated complexity. Not zero, probably never zero, but smaller every day. If you reasonably can get away with deploying a simpler kind of storage, I recommend it.
Re: A Ceph war story
#35Re: A Ceph war story
#36Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…
Interesting read and helpful lesson list. I've only used Ceph as provided to be by others and considered setting it up in some instances. Didn't know about the development of BlusStore and it does seem much simpler. The choice between xfs, btrfs, ext4 always seemed a bit unclear (except that I had experienced non-Ceph troubles with btrfs). Note to self: use ceph-volume/BlueStore.
Re: A Ceph war story
#37Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…
>Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf This gave me a concern. My kube nodes do use XFS in some cases but Ceph uses raw block devices. So XFS is only used for system files, not for Ceph. Except of course to store Ceph config on each node. So…
Re: A Ceph war story
#38Earlier quoted context omitted.
Ceph user here. Ceph works fine 99.9% of the time until it doesn't. In this case you get a bit panic and start to google, ask in IRC for help and take a look at the bug tracker. In all critical cases of the past we were able to recover and gained ceph experience. I guess this is how you become a ceph expert.
Curious about Ceph, what feature are you using it for? Would it make sense to serve big number of websites, stored on 20 servers, using single Ceph filesystem with some SSD cache? Or am I better off having just 20 independent servers?
Mainly CephFS. It allows you similar to NFS to have a big shared filesystem over network. If you have clients that require access to a triple digit terabyte or petabyte sized filessystem, then you might want to consider ceph.
> Or am I better off having just 20 independent servers?
If you have 20 independent websites, then I would recommend to provide them their own storage. That is because if your ceph instance has a downtime, then all your 20 websites would be down simultaneously.
Re: A Ceph war story
#39Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…
>Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf This gave me a concern. My kube nodes do use XFS in some cases but Ceph uses raw block devices. So XFS is only used for system files, not for Ceph. Except of course to store Ceph config on each node. So…
- Filestore is the legacy backend that uses files on a filesystem (strongly recommended to be XFS)
- Bluestore is the modern backend that uses raw device files directly
Re: A Ceph war story
#40This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.
> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…