A Ceph war story • Hacker News Board

Re: A Ceph war story

#21

Post by rjzzleep » Sat, Apr 10, 2021, 10:41 AM UTC

sekh60 wrote: post #18

ps wrote: post #14

Earlier quoted context omitted.

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k... “One of the issues is that the officially recommended way to install it seems to be rook-ceph wi…

You are correct about cephadm being the recommended install method. For NVMe OSDs how many daemons are using the same drive? Normally to get performance you have to colocate two or four osd daemons on a single NVMe device. Also ceph benefits the most from massive parallelism, how many clients do you have? 4 OSDd is not a lot to spread the load over, even if the drives are lightning fast. There is a new io pathway in…

Cephadm is very new. So yes it may be the recommended solution NOW. Precisely because all previous solutions were very fragile. I don’t have any experience with it for these reasons.

Somehow Linux new filesystems take very long from initial release to reliable use. As seen also with btrfs.

Maybe filesystems are just hard I don’t know but somehow the solutions seem very Hacky.

Re: A Ceph war story

#22

Post by Proven » Sat, Apr 10, 2021, 11:18 AM UTC

bsder wrote: post #11

fulafel wrote: post #2

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

> I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster.

Have you ever worked with decent enterprise hardware?

Re: A Ceph war story

#23

Post by luma » Sat, Apr 10, 2021, 11:46 AM UTC

bsder wrote: post #11

fulafel wrote: post #2

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

RAID controllers are great if you're using them to do the thing that they do. If you are going to host a software defined storage solution on top, then the thing RAID controllers do for you (abstract the hardware storage devices into a block volume device) becomes counter productive.

Most modern array controllers can be switched to HBA mode, which disables the RAID bits entirely and passes SAS/SATA commands through directly to the device. The article doesn't say if this was done, but based on the description of events, it wasn't.

Running software defined storage on top of hardware RAID functionality is begging for trouble.

Re: A Ceph war story

#24

Post by trebligdivad » Sat, Apr 10, 2021, 1:03 PM UTC

Why didn't they just roll back to the old kernel before doing anything else?

Re: A Ceph war story

#25

Post by mikagrml » Sat, Apr 10, 2021, 1:21 PM UTC

trebligdivad wrote: post #24

Why didn't they just roll back to the old kernel before doing anything else?

(Author here) Because it was absolutely unclear at that time yet, that an older kernel version could mount the XFS partition but the newer kernel versions could not (this only came up during the post mortem/RCA later). Furthermore the clock skew and mon_host issues gave us a wrong picture of the situation. (Also as the hosts are running as hypervisor systems, the kernel version should ~match with the environment (Proxmox/KVM), so there might be other/unclear risks with running such a setup.)

Re: A Ceph war story

#26

Post by mikagrml » Sat, Apr 10, 2021, 1:34 PM UTC

terom wrote: post #6

Apart from the NTP tangent, this sounds like a Linux XFS / ServeRAID M5210 firmware issue. Your XFS filesystems created using the incorrect block/io sizes reported by the RAID controller would have been unmountable on the newer Linux kernel regardless of Ceph. Lesson learned: your configuration management also needs to control for firmware versions such that the same issue would have shown up in a dev/test environmen…

(Author here) Yes, it was an XFS/controller issue, but Ceph reported the failure. :) (IMO it wasn't really a good decision from Ceph to use 100MB XFS partitions as a kind of database, but nowadays ceph-disk (which uses those XFS partitions) is gone, and instead ceph-volume uses a different approach via LVM.)

Regarding configuration management/firmware version: yes - especially, as you'd need to also rebuild disks in the dev/test environment with the identical configuration (firmware, disks,...), to ensure it's actually identical. And even if we neglect load/capacity/usage issues (problems might show up only under specific work loads), there are also further "invisible" layers/components like cables, NICs, switches,… and their firmware versions which are also relevant. Not exactly trivial. :)

Re: A Ceph war story

#27

Post by _mikulely » Sat, Apr 10, 2021, 1:36 PM UTC

emmericp wrote: post #15

Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…

ceph-volume still relies on LVM, which brings unnecessary complexity.

We'd like to stick to ceph-disk(already unavailable in the P release) with raw block device only.

Re: A Ceph war story

#28

Post by INTPenis » Sat, Apr 10, 2021, 1:43 PM UTC

emmericp wrote: post #15

Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one. Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode. Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems…

>Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

This gave me a concern. My kube nodes do use XFS in some cases but Ceph uses raw block devices. So XFS is only used for system files, not for Ceph. Except of course to store Ceph config on each node.

So I assume I'm safe. I'm not entirely sure how you'd use XFS with Ceph because Ceph uses a raw device file and formats it for its own storage.

Re: A Ceph war story

#29

Post by effie » Sat, Apr 10, 2021, 1:43 PM UTC

bsder wrote: post #11

fulafel wrote: post #2

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to p…

> I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster.

1) inertia 2) simpler way to get up and running for people who are not skilled in CLI or other SW RAID tools (HW RAID controllers have nice BIOS setup) 3) HW RAID BIOS setup always boots and is accessible even if OS can't boot 4) HW RAID stays the same for decades so SW updates are less likely to screw something

Re: A Ceph war story

#30

Post by antongribok » Sat, Apr 10, 2021, 1:46 PM UTC

Tepix wrote: post #3

Excellent writeup and impressive analysis! Are many people using Ceph and what are your experiences like?

I (with 2-3 other people) run over 100 PB on Ceph clusters in production, supporting some critical functions of a Fortune 50 enterprise.

We use Ceph for block and object workloads (no CephFS). Most of our clusters are still on Luminous (v12) and Filestore (XFS), and only our newer clusters that are being built now are on Nautilus (v14) Bluestore. We plan on migrating to Bluestore this year (and likely next year).

I'm on-call basically all the time, but we'll have maybe 1 issue per year where we have to act immediately. Most failures that would happen on a Saturday can wait until Monday to be acted on.

The smallest cluster that we'll build is 6 nodes. Our largest cluster now is 120 nodes with 1920 OSDs. We might build a larger cluster this year.

Back in Ceph Hammer days, I had a 6-node cluster lose 3 nodes, one node at a time, over a course of a couple of days with zero downtime.

We deploy with Ansible, and have our own, very paranoid and opinionated playbook for doing a rolling cluster reboot or restart, that checks multiple things before moving on to the next node.