Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

51–60 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#51

Post by jeffbee »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

I think the chief source of inefficiency in this architecture would be the NVMe controller. When the operating system and the NVMe device are at arm's length, there is natural inefficiency, as the controller needs to infer the intent of the request and do its best in terms of placement and wear leveling. The new FDP (flexible data placement) features try to address this by giving the operating system more control. The best thing would be to just hoist it all up into the host operating system and present the flash, as nearly as possible, as a giant field of dumb transistors that happens to be a PCIe device. With layers of abstraction removed, the hardware unit could be something like an Atom with integrated 100gbps NICs and a proportional amount of flash to achieve the desired system parallelism.

Re: Ceph: A Journey to 1 TiB/s

#52

Post by booi »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

Is that a lot of overhead? The disk itself uses about 10W and high speed controllers use about 75W leaves pretty much 100W for the rest of the system including overhead of about 10%. Scale up the system to 16 disks and there’s not a lot of room for improvement

Re: Ceph: A Journey to 1 TiB/s

#54

Post by kbenson »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

There probably is a sweet spot for power to speed, but I think it's possibly a bit larger than you suggest. There's overhead from the other components as well. For example, the Mellanox NIC seems to utilize about 20W itself, and while the reduced numbers of drives might allow for a single port NIC which seems to use about half the power, if we're going to increase the number of cables (3 per 12 disks instead of 2 per 5), we're not just increasing the power usage of the nodes themselves put also possible increasing the power usage or changing the type of switch required to combine the nodes.

If looked at as a whole, it appears to be more about whether you're combining resources at a low level (on the PCI bus on nodes) or a high level (in the switching infrastructure), and we should be careful not to push power (or complexity, as is often a similar goal) to a separate part of the system that is out of our immediate thoughts but still very much part of the system. Then again, sometimes parts of the system are much better at handling the complexity for certain cases, so in those cases that can be a definite win.

Re: Ceph: A Journey to 1 TiB/s

#55

Post by CyberDildonics »

KeplerBoy wrote: post #3
riku_iki wrote: post #2

What router/switch one would use for such speed?

800Gbps via OSFP and QSFP-DD are already a thing. Multiple vendors have NICs and switches for that.

16x PCIe 4.0 is 32GB/s 16x PCIe 5.0 should be 64 GB/s, how is any computer using 100 GB/s ?

Re: Ceph: A Journey to 1 TiB/s

#56

Post by alberth »

Ceph has an interesting history.

It was created at Dreamhost (DH), for their internal needs by the founders.

DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers).

They spun Ceph off and Redhat bought it.

https://en.wikipedia.org/wiki/DreamHost

Re: Ceph: A Journey to 1 TiB/s

#57

Post by KeplerBoy »

_zoltan_ wrote: post #25
KeplerBoy wrote: post #3

Earlier quoted context omitted.

800Gbps via OSFP and QSFP-DD are already a thing. Multiple vendors have NICs and switches for that.

can you show me a 800G NIC? the switch is fine, I'm buying 64x800G switches, but NIC wise I'm limited to 400Gbit.

fair enough, it seems I was mistaken about the NIC. I guess that has to wait for PCIe 6 and should arrive soon-ish.

Re: Ceph: A Journey to 1 TiB/s

#58

Post by KeplerBoy »

CyberDildonics wrote: post #55
KeplerBoy wrote: post #3

Earlier quoted context omitted.

800Gbps via OSFP and QSFP-DD are already a thing. Multiple vendors have NICs and switches for that.

16x PCIe 4.0 is 32GB/s 16x PCIe 5.0 should be 64 GB/s, how is any computer using 100 GB/s ?

I was talking about Gigabit/s, not Gigabyte/s.

The article however actually talks about Terabyte/s scale, albeit not over a single node.

Re: Ceph: A Journey to 1 TiB/s

#59

Post by CyberDildonics »

KeplerBoy wrote: post #58
CyberDildonics wrote: post #55

Earlier quoted context omitted.

16x PCIe 4.0 is 32GB/s 16x PCIe 5.0 should be 64 GB/s, how is any computer using 100 GB/s ?

I was talking about Gigabit/s, not Gigabyte/s. The article however actually talks about Terabyte/s scale, albeit not over a single node.

800 gigabits is 100 gigabytes which is still more than PCIe 5.0 16x 64 gigabyte per second bandwidth.

You said there were 800 gigabit network cards, I'm wondering how that much bandwidth makes it to the card in the first place.

The article however actually talks about Terabyte/s scale, albeit not over a single node.

This does not have anything to do with what you originally said, you were talking about 800gb single ports.

Re: Ceph: A Journey to 1 TiB/s

#60

Post by Palomides »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

here's a weird calculation:

this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts

a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data

so OP's cluster, back of the envelope, is basically the same bits per second per watt that a very efficient arm system can do

I don't think running tiny nodes would actually get you any more efficiency, and would probably cost more! performance per watt is quite good on powerful servers now

anyway, this is all open source software running on off-the-shelf hardware, you can do it yourself for a few hundred bucks

Post reply on HN