Does someone knows how Ceph compares to other object storage engine like MinIO/Garage/...?
I would love to see some benchmarks there.
Ceph: A Journey to 1 TiB/s
171–180 of 216 posts
Re: Ceph: A Journey to 1 TiB/s
#172I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…
IIRC, WD has experimented with placing Ethernet and some compute directly onto hard drives some time back.
sigh I used to do some small-scale Ceph back in 2017 or so...
Re: Ceph: A Journey to 1 TiB/s
#173Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…
Curious, what do you mean by "know what you go into" re glusterfs?
I recently tried ceph in a homelab setup, gave up because of complexity, and settled on glusterfs. I'm not a pro though, so I'm not sure if there's any shortcomings that are clear to everybody but me, hence why your comment caught my attention.
Re: Ceph: A Journey to 1 TiB/s
#174I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…
here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…
I checked selling prices of those racks + top end SSDs, this 1Tb/s achievement runs on $4 million worth of hardware cluster. Or more I didn't check the networking interface costs.
But yeah could run on commodity hardware. Not sure those highly efficient arm packaged for a premium from Apple would beat the Dell racks though regarding throughput relative to hardware investment costs.
Re: Ceph: A Journey to 1 TiB/s
#175Earlier quoted context omitted.
I have always wanted to set up a ceph system with one drive per node. The ideal form factor would be a drive with a couple network interfaces built in. western digital had a press release about an experiment they did that was exactly this, but it never ended up with drive you could buy. The hardkernel HC2 SOC was a nearly ideal form factor for this, and I still have a stack of them laying around that I bought to make…
Sounds like the Seagate Kinetic HDD: https://www.seagate.com/www-content/product-content/hdd-fam/...
That would be perfect. Unfortunately, going by the data sheet it would not run ceph you would have to work with seagate's proprietary object store. I will note that as far as I can tell it is unobtainium. none of the usual vendors stock them, you probably have to prove to seagate that you are a "serious enterprise customer" and commit to a thousand units before they will let you buy some.
Re: Ceph: A Journey to 1 TiB/s
#176Earlier quoted context omitted.
here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…
Trusting your maths, damn Apple did a great job on their M design.
Didn't ARM (the company, that originally designed ARM processors) do most of that job and Apple pushed perf to consumption even further?
Re: Ceph: A Journey to 1 TiB/s
#177Earlier quoted context omitted.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…
I'd throw minio [1] in the list there as well for homelab k8s object storage. [1] https://min.io/
Minio doesn't make any sense to me in a homelab. Unless I'm reading it wrong it sounds like a giant pain to add more capacity while it is already in use. There's basically no situation where I'm more likely to add capacity over time than a homelab.
Re: Ceph: A Journey to 1 TiB/s
#178Earlier quoted context omitted.
here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…
I checked selling prices of those racks + top end SSDs, this 1Tb/s achievement runs on $4 million worth of hardware cluster. Or more I didn't check the networking interface costs. But yeah could run on commodity hardware. Not sure those highly efficient arm packaged for a premium from Apple would beat the Dell racks though regarding throughput relative to hardware investment costs.
Dell’s list prices have essentially nothing to do with the prices that any competent buyer would actually pay, especially when storage is involved. Look at the prices of Dell disks, which are nothing special compared to name brand disks of equal or better spec and much lower list price.
I don’t know what discount large buyers get, but I wouldn’t be surprised if it’s around 75%.
Re: Ceph: A Journey to 1 TiB/s
#179Does someone knows how Ceph compares to other object storage engine like MinIO/Garage/...? I would love to see some benchmarks there.
This would be great, to have a universal benchmark of all available open source solutions for self-hosting. Links appreciated!
Re: Ceph: A Journey to 1 TiB/s
#180Earlier quoted context omitted.
I think the Mac Mini has massively more compute than needed for this kind of work. It also has a power supply, and computer power supplies are generally not amazing at low output. I’m imagining something quite specialized. Use a low frequency CPU with either vector units or even DMA engines optimized for the specific workloads needed, or go all out and arrange for data to be DMAed directly between the disk and the NI…
sounds like a DPU (mellanox bluefield for example), they're entire ARM systems with a high speed NIC all on a PCIe card, I think the bluefield ones can even directly interface over the bus to nvme drives without the host system involved
That Bluefield hardware looks neat, although it also sounds like a real project to program it :).
I can imagine two credible configurations for high efficiency:
1. A motherboard with a truly minimal CPU for bootstrapping but a bit beefy PCIe root complex. 32 lanes to the DPU and a bunch of lanes for NVMe. The CPU doesn’t touch the data at all. I wonder if anyone makes a motherboard optimized like this — a 64-lane mobo with a Xeon in it would be quite wasteful but fine for prototyping I suppose.
2. Wire up the NVMe ports directly to the Bluefield DPU, letting the DPU be the root complex. At least 28 of the lanes are presumably usable for this or maybe even all 32. It’s not entirely clear to me that the Bluefield DPU can operate without a host computer, though.