41–50 of 216 posts
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
Earlier quoted context omitted.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…
I'd throw minio [1] in the list there as well for homelab k8s object storage. [1] https://min.io/
I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
I'm about 1/2 through the process of moving my 15 virtual machines over. It is a little slow but tolerable. Not having to decide on RAIDs or a NAS ahead of time is amazing. I can throw disks and nodes at it whenever.
I used to love doing experiments like this. I was afforded that luxury as a tech lead back when I was at Cisco setting up Kubernetes on bare metal and getting to play with setting up GlusterFS and Ceph just to learn and see which was better. This was back in 2017/2018 if I recall. Good ole days. Loved this writeup!
This has happened in multiple clusters, using rook/ceph as well as Longhorn.
Earlier quoted context omitted.
For those curious, a "bargain" on a 100gbps switch means about $1350
On a cluster with more than $1M of NVMe disks, that does actually seem like a bargain. (Note that the linked MikroTik switch only has 100gbe on a few ports, and wouldn't really classify as a full 100gbe switch to most people)
That was more for the benefit of others like myself, who were wondering if "bargain" was comparative, or inexpensive enough that it might be worth buying one next time they upgraded switches. For me personally it's still an order of magnitude away from that.
Earlier quoted context omitted.
I really wish there was a benchmark comparing all of these + MinIO and S3. I'm in the market for a key value store, using S3 for now but eyeing moving to my own hardware in the future and having to do all the work to compare these is one of the major things making me procrastinate.
minio is good but you really need fast disks. They also really don't like, when you want to change the size of your cluster setup. No plan to add cache disks, they just say use faster disks. I have it running, goes smoothly but not really user friendly to optimize
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…