Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

181–190 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#181

Post by cholmon »

nullwarp wrote: post #5
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

GlusterFS support looks to be permanently ending later this year.

https://access.redhat.com/support/policy/updates/rhs

Note that the Red Hat Gluster Storage product has a defined support lifecycle through to 31-Dec-24, after which the Red Hat Gluster Storage product will have reached its EOL. Specifically, RHGS 3.5 represents the final supported RHGS series of releases.

For folks using GlusterFS currently, what's your plan after this year?

Re: Ceph: A Journey to 1 TiB/s

#182

Post by qingcharles »

plagiarist wrote: post #163
qingcharles wrote: post #102

Earlier quoted context omitted.

That reminds me of a calculation I did which showed that my desktop PC would be more powerful than all of the computers on the planet combined in like 1978 :D

My phone has more computation than anything I would have imagined owning, and I sometimes turn on the screen just to use as a quick flashlight.

Haha.. imagine taking it back to 1978 and showing how it has more computing power than the entire planet and then telling them that you mostly just use it to find that thing you lost under the couch :D

Re: Ceph: A Journey to 1 TiB/s

#183

Post by reactordev »

plagiarist wrote: post #177
reactordev wrote: post #11

Earlier quoted context omitted.

I'd throw minio [1] in the list there as well for homelab k8s object storage. [1] https://min.io/

Minio doesn't make any sense to me in a homelab. Unless I'm reading it wrong it sounds like a giant pain to add more capacity while it is already in use. There's basically no situation where I'm more likely to add capacity over time than a homelab.

You get a new nas (minio server pool) and you plug it into your home lab (site replication) and now it's part of the distributed minio storage layer (k8s are happy). How is that hard? It's the same basic thing for Ceph or any distributed JBOD mass storage engine. Minio has some funkiness with how you add more storage but it's totally capable of doing it while in use. Everything is atomic.

Re: Ceph: A Journey to 1 TiB/s

#184

Post by plagiarist »

peter_d_sherman wrote: post #83

Ceph is interesting... open source software whose only purpose is to implement a distributed file system... Functionally, Linux implements a file system (well, several!) as well (in addition to many other OS features) -- but (usually!) only on top of local hardware. There seems to be some missing software here -- if we examine these two paradigms side-by-side. For example, what if I want a Linux (or more broadly, a g…

It sounds you want microkernels, and I agree, it would be nice.

Re: Ceph: A Journey to 1 TiB/s

#185

Post by ComputerGuru »

amadio wrote: post #162
theyinwhy wrote: post #129

Earlier quoted context omitted.

Great! What's your take on ceph? Is the idea to migrate to EOS long term?

EOS and ceph have different use cases at CERN. EOS holds physics data and user data in CERNBox, while ceph is used for a lot of the rest (e.g. storage for VMs, and other applications). So both will continue to be used as they are now. CERN has over 100PB on ceph.

Is there a reason you run both and don't converge on one or the other?

Re: Ceph: A Journey to 1 TiB/s

#186

Post by georgyo »

Palomides wrote: post #60
amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…

You're comparing one machine with many machines.

You're comparing raw disks with shards and erasure encouraging.

Lastly, you're comparing only network bandwidth and not storage capacity.

Re: Ceph: A Journey to 1 TiB/s

#187

Post by nh2 »

loeg wrote: post #6
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Why would you bother with a distributed filesystem when you don't have to?

One reason for using Ceph instead of other RAID solutions on a single machine is that it supports disk failures more flexibly.

In most RAIDs (including ZFS's, to my knowledge), the set of disks that can fail together is static.

Say you have physical disks A B C D E F; common setup is to group RAID1'd disks into a pool such as `mirror(A, B) + mirror(C, D) + mirror(E, F)`.

With that, if disk A fails, and then later B fails before you replace A, your data is lost.

But with Ceph, and replication `size = 2`, when A fails, Ceph will (almost) immediately redistribute your data so that it has 2 replicas again, across all remaining disks B-F. So then B can fail and you still have your data.

So in Ceph, you give it a pool of disks and tell it to "figure out the replication" iself. Most other systems don't offer that; the human defines a static replication structure.

Re: Ceph: A Journey to 1 TiB/s

#190

Post by BlackLotus89 »

speedgoose wrote: post #42
reactordev wrote: post #11

Earlier quoted context omitted.

I'd throw minio [1] in the list there as well for homelab k8s object storage. [1] https://min.io/

Also garage. https://garagehq.deuxfleurs.fr/

Garage seems to only to duplication https://garagehq.deuxfleurs.fr/documentation/design/goals/

> Storage optimizations: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication.

This is probably a nogo for most use cases where you work with large datasets....

Post reply on HN