Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

1–10 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#5

Post by nullwarp »

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother.

When it works, it works great - when it goes wrong it's a huge headache.

Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup:

- seaweedfs has been rock solid for me for years in both small and huge scales. we actually moved our production ceph setup to this.

- longhorn was solid for me when i was in the k8s world

- glusterfs is still fine as long as you know what you are going into.

Re: Ceph: A Journey to 1 TiB/s

#8

Post by ianlevesque »

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

I played around with it and it has a very cool web UI, object storage & file storage, but it was very hard to get decent performance and it was possible to get the metadata daemons stuck pretty easily with a small cluster. Ultimately when the fun wore off I just put zfs on a single box instead.

Re: Ceph: A Journey to 1 TiB/s

#9

Post by epistasis »

riku_iki wrote: post #2

What router/switch one would use for such speed?

Given their configuration of just 4U spread across 17 racks, there's likely a bunch of compute in the rest of the rack, and 1-2 top of rack switches like this:

https://www.qct.io/product/index/Switch/Ethernet-Switch/T700...

And then you connect the TOR switches to higher level switches in something like a Clos distribution to get the desired bandwidth between any two nodes:

https://www.techtarget.com/searchnetworking/definition/Clos-...

Re: Ceph: A Journey to 1 TiB/s

#10

Post by NavinF »

riku_iki wrote: post #2

What router/switch one would use for such speed?

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work.

Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

Post reply on HN