71–80 of 216 posts
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
I had 4 NUCs running Proxmox+Ceph for a few years, and apart from slightly annoying slowness syncing after spinning the machines up from cold start, it all ran very smoothly.
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…
My requirements for a storage solution are:
> Single root file system
> Storage device failure tolerance
> Gradual expansion capability
The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have the money to buy 200 hard disks all at once. I need to gradually expand capacity as needed.
I was attracted to this ceph because it apparently allows you to throw a bunch of drives of any make and model at it and it just pools them all up without complaining. The complexity is nightmarish though.
ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. Expansion features seem to be just about to land for quite a few years now. I remember getting excited about it after seeing news here only for people to deflate my expectations. Btrfs has a flexible block allocator which is just what I need but... It's btrfs.
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
Ceph has an interesting history. It was created at Dreamhost (DH), for their internal needs by the founders. DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers). They spun Ceph off and Redhat bought it. https://en.wikipedia.org/wiki/DreamHost
A bit more to the story is that it was created also at UC Santa Cruz, by Sage Weil, a Dreamhost founder, while he was doing graduate work there. UCSC has had a lot of good storage research.
Where can I read about the rationale for ceph as a project? I'm not familiar with it.
The downside is durability and operations - we have to keep Ceph alive and are responsible for making sure the data is persistent. That said, we're storing cache from container builds, so in the worst-case where we lose the storage cluster, we can run builds without cache while we restore.
Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.
Related question, how does someone get into working with Ceph? Other than working somewhere that already uses it.
https://docs.ceph.com/en/latest/cephadm/
To learn about Ceph, I recommend you create at least 3 KVM virtual machines (using virt-manager) on a development box, network them together, and use cephadm to set up a cluster between the VMs. The RAM and storage requirements aren't huge (Ceph can run on Raspberry Pis, after all) and I find it a lot easier to figure things out when I have a desktop window for every node.
I recently set up Ceph twice. Now that Ceph (specifically RBD) is providing the storage for virtual machines, I can live-migrate VMs between hosts and reboot hosts (with zero guest downtime) anytime I need. I'm impressed with how well it works.
Earlier quoted context omitted.
There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…
I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…
if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured.
[1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
Ceph has an interesting history. It was created at Dreamhost (DH), for their internal needs by the founders. DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers). They spun Ceph off and Redhat bought it. https://en.wikipedia.org/wiki/DreamHost
I think it was the university project of one of the founders, and the others jumped in supporting it. Docker has a similar origins story as far as I know.