Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
I think you need 3 or was it 5 machines?
proxmox will use it - just click to install
Ceph: A Journey to 1 TiB/s
21–30 of 216 posts
Re: Ceph: A Journey to 1 TiB/s
#22Earlier quoted context omitted.
> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?
any switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'
I will forever be scarred by the "Gigabit" switches of old that were 2 gigabit ports and 22 100mb ports. Coworker bought it missing the nuance.
Re: Ceph: A Journey to 1 TiB/s
#23Earlier quoted context omitted.
Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work. Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases
> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?
Even the bargain Mikrotik can do 1.2Tbps https://mikrotik.com/product/crs518_16xs_2xq
Re: Ceph: A Journey to 1 TiB/s
#24Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
I have some experience with Ceph, both for work, and with homelab-y stuff.
First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes.
For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines.
Also, Ceph does prefer physical access to disks (similar to ZFS).
And you do need decent networking connectivity - I think that's the main thing people think of, when they think of high hardware requirements for Ceph. Ideally 10Gbe at the minimum - although more if you want higher performance - there can be a lot of network traffic, particularly with things like backfill. (25Gbps if you can find that gear cheap for homelab - 50Gbps is a technological dead-end. 100Gbps works well).
But honestly, for a homelab, a cheap mini PC or NUC with 10Gbe will work fine, and you should get acceptable performance, and it'll be good for learning.
You can install Ceph directly on bare-metal, or if you want to do the homelab k8s route, you can use Rook (https://rook.io/).
Hope this helps, and good luck! Let me know if you have any other questions.
Re: Ceph: A Journey to 1 TiB/s
#25Re: Ceph: A Journey to 1 TiB/s
#26Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
There's a blog post they did where they setup Ceph on some rPI 4's. I'd say that's not significant hardware at all. [1] [1] https://ceph.io/en/news/blog/2022/install-ceph-in-a-raspberr...
I think "significant" turns out to mean the number of nodes required.
Re: Ceph: A Journey to 1 TiB/s
#27Earlier quoted context omitted.
any switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'
I will forever be scarred by the "Gigabit" switches of old that were 2 gigabit ports and 22 100mb ports. Coworker bought it missing the nuance.
Still happens, gotta see if the top speed mentioned is an uplink or normal ports.
Re: Ceph: A Journey to 1 TiB/s
#28Earlier quoted context omitted.
> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?
Even the bargain Mikrotik can do 1.2Tbps https://mikrotik.com/product/crs518_16xs_2xq
For those curious, a "bargain" on a 100gbps switch means about $1350
Re: Ceph: A Journey to 1 TiB/s
#29Re: Ceph: A Journey to 1 TiB/s
#30I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all.
I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V CPU. Perhaps an eMMC chip or SD slot for boot.
This could scale down to just a few nodes, and it reduces the exposure to a single failure taking out 10 disks at a time.
I bet a lot of copies of this system could fit in a 4U enclosure. Optionally the same enclosure could contain two entirely independent switches to aggregate the internal nodes.