Ceph: A Journey to 1 TiB/s

Post by progval » Sat, Jan 20, 2024, 8:14 AM UTC

somat wrote: post #109

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

I have always wanted to set up a ceph system with one drive per node. The ideal form factor would be a drive with a couple network interfaces built in. western digital had a press release about an experiment they did that was exactly this, but it never ended up with drive you could buy. The hardkernel HC2 SOC was a nearly ideal form factor for this, and I still have a stack of them laying around that I bought to make…

Post by _zoltan_ » Sat, Jan 20, 2024, 9:10 AM UTC

mrb wrote: post #91

wmf wrote: post #87

Earlier quoted context omitted.

I think PCIe TLP overhead and NVMe commands account for the difference between 7 and 8 GB/s.

You are probably right. Reading some old notes of mine when I was fine-tuning PCIe bandwith on my ZFS server, I had discovered back then that a PCIe Max_Payload_Size of 256 bytes limited usable bandwidth to about 74% of the link's theoretical max. I had calculated that 512 and 1024 bytes (the maximum) would raise it to respectively about 86% and 93% (but my SATA controllers didn't support a value greater than 256.)

Post by eurekin » Sat, Jan 20, 2024, 9:32 AM UTC

progval wrote: post #131

somat wrote: post #109

Earlier quoted context omitted.

I have always wanted to set up a ceph system with one drive per node. The ideal form factor would be a drive with a couple network interfaces built in. western digital had a press release about an experiment they did that was exactly this, but it never ended up with drive you could buy. The hardkernel HC2 SOC was a nearly ideal form factor for this, and I still have a stack of them laying around that I bought to make…

I used to use Ceph Luminous (v12) on these, they worked fine. Unfortunately, a bug in Nautilus (v14) prevented 32-bits and 64-bits archs from talking to each other. Pacific (v16) allegedly solves this, but I didn't try it: https://ceph.com/en/news/blog/2021/v16-2-5-pacific-released/ If you want to try it with a more modern (and 64-bits) device, the hardkernel HC4 might do it for you. It's conceptually similar to the…

Post by imiric » Sat, Jan 20, 2024, 9:42 AM UTC

loeg wrote: post #64

imiric wrote: post #32

Earlier quoted context omitted.

For the same reason you would use one in enterprise deployments: if setup properly, it's easier to scale. You don't need to invest in a huge storage server upfront, but could build it out as needed with cheap nodes. Assuming it works painlessly as a single node filesystem, of which I'm not yet convinced if the existing solutions do.

> if setup properly, it's easier to scale For home use/needs, I think vertical scaling is much easier.

Post by magicalhippo » Sat, Jan 20, 2024, 10:05 AM UTC

mrb wrote: post #132

magicalhippo wrote: post #121

Earlier quoted context omitted.

They're benchmarking random IO though, and the disks can "only" do a bit over 1000k random 4k read IOPS, which translates to about 5 GiB/s. With 320 OSDs thats around 1.6 TiB/s. At least thats the number I could find. Not exactly tons of reviews on these enterprise NVMe disks... Still, that seems like a good match to the NICs. At this scale most workloads will likely appear as random IO at the storage layer anyway.

The benchmark were they accomplish 1025 GiB/s is for sequential reads. For random reads they do 25.5M iops or ~100 GiB/s. See last table, column "630 OSDs (3x)".