Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

11–20 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#11

Post by reactordev »

nullwarp wrote: post #5
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I'd throw minio [1] in the list there as well for homelab k8s object storage.

[1] https://min.io/

Re: Ceph: A Journey to 1 TiB/s

#12

Post by iwontberude »

loeg wrote: post #6
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Why would you bother with a distributed filesystem when you don't have to?

It's cool to cluster everything for some people (myself included). I see it more like a design constraint than a pure benefit.

Re: Ceph: A Journey to 1 TiB/s

#13

Post by dataangel »

nullwarp wrote: post #5
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I really wish there was a benchmark comparing all of these + MinIO and S3. I'm in the market for a key value store, using S3 for now but eyeing moving to my own hardware in the future and having to do all the work to compare these is one of the major things making me procrastinate.

Re: Ceph: A Journey to 1 TiB/s

#14

Post by riku_iki »

NavinF wrote: post #10
riku_iki wrote: post #2

What router/switch one would use for such speed?

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work. Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

> So any 100G pizza box switches should work.

but will it be able to handle combined TB/s traffic?

Re: Ceph: A Journey to 1 TiB/s

#15

Post by stuff4ben »

I used to love doing experiments like this. I was afforded that luxury as a tech lead back when I was at Cisco setting up Kubernetes on bare metal and getting to play with setting up GlusterFS and Ceph just to learn and see which was better. This was back in 2017/2018 if I recall. Good ole days. Loved this writeup!

Re: Ceph: A Journey to 1 TiB/s

#16

Post by reactordev »

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

There's a blog post they did where they setup Ceph on some rPI 4's. I'd say that's not significant hardware at all. [1]

[1] https://ceph.io/en/news/blog/2022/install-ceph-in-a-raspberr...

Re: Ceph: A Journey to 1 TiB/s

#17

Post by SteveNuts »

bluedino wrote: post #7
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Related question, how does someone get into working with Ceph? Other than working somewhere that already uses it.

You could start by installing Proxmox on old machines you have, it uses Ceph for its distributed storage, if you choose to use it.

Re: Ceph: A Journey to 1 TiB/s

#18

Post by baq »

riku_iki wrote: post #14
NavinF wrote: post #10

Earlier quoted context omitted.

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work. Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?

any switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'

Re: Ceph: A Journey to 1 TiB/s

#19

Post by aaronax »

riku_iki wrote: post #14
NavinF wrote: post #10

Earlier quoted context omitted.

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work. Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

> So any 100G pizza box switches should work. but will it be able to handle combined TB/s traffic?

Yes. Most network switches can handle all ports at 100% utilization in both directions simultaneously.

Take for example the Mellanox SX6790 available for less than $100 on eBay. It has 36 56gbps ports. 36 * 2 * 56 = 4032gbps and it is stated to have a switching capacity of 4.032Tbps.

Edit: I guess you are asking how one would possibly sip 1TiB/s of data into a given client. You would need multiple clients spread across several switches to generate such load. Or maybe some freaky link aggregation. 10x 800gbps links for your client, plus at least 10x 800gbps links out to the servers.

Re: Ceph: A Journey to 1 TiB/s

#20

Post by candiddevmike »

bluedino wrote: post #7
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Related question, how does someone get into working with Ceph? Other than working somewhere that already uses it.

Look into the Rook project
Post reply on HN