Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

61–70 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#61

Post by sgarland »

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Yes. I first tried it with Rook, and that was a disaster, so I shifted to Longhorn. That has had its own share of problems, and is quite slow. Finally, I let Proxmox manage Ceph for me, and it’s been a dream. So far I haven’t migrated my K8s workloads to it, but I’ve used it for RDBMS storage (DBs in VMs), and it works flawlessly.

I don’t have an incredibly great setup, either: 3x Dell R620s (Ivy Bridge-era Xeons), and 1GBe. Proxmox’s corosync has a dedicated switch, but that’s about it. The disks are nice to be fair - Samsung PM863 3.84 TB NVMe. They are absolutely bottlenecked by the LAN at the moment.

I plan on upgrading to 10GBe as soon as I can convince myself to pay for an L3 10G switch.

Re: Ceph: A Journey to 1 TiB/s

#62

Post by epistasis »

alberth wrote: post #56

Ceph has an interesting history. It was created at Dreamhost (DH), for their internal needs by the founders. DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers). They spun Ceph off and Redhat bought it. https://en.wikipedia.org/wiki/DreamHost

A bit more to the story is that it was created also at UC Santa Cruz, by Sage Weil, a Dreamhost founder, while he was doing graduate work there. UCSC has had a lot of good storage research.

Re: Ceph: A Journey to 1 TiB/s

#63

Post by Palomides »

margalabargala wrote: post #28
bombcar wrote: post #23

Earlier quoted context omitted.

Even the bargain Mikrotik can do 1.2Tbps https://mikrotik.com/product/crs518_16xs_2xq

For those curious, a "bargain" on a 100gbps switch means about $1350

there's usually some used dx010 (32x100gbe) on ebay for less than $500

the cheapest new 100gbe switch I know of is the mikrotik CRS504-4XQ-IN (4x100gbe, around $650)

Re: Ceph: A Journey to 1 TiB/s

#64

Post by loeg »

imiric wrote: post #32
loeg wrote: post #6

Earlier quoted context omitted.

Why would you bother with a distributed filesystem when you don't have to?

For the same reason you would use one in enterprise deployments: if setup properly, it's easier to scale. You don't need to invest in a huge storage server upfront, but could build it out as needed with cheap nodes. Assuming it works painlessly as a single node filesystem, of which I'm not yet convinced if the existing solutions do.

> if setup properly, it's easier to scale

For home use/needs, I think vertical scaling is much easier.

Re: Ceph: A Journey to 1 TiB/s

#65

Post by bombcar »

margalabargala wrote: post #48
epistasis wrote: post #39

Earlier quoted context omitted.

On a cluster with more than $1M of NVMe disks, that does actually seem like a bargain. (Note that the linked MikroTik switch only has 100gbe on a few ports, and wouldn't really classify as a full 100gbe switch to most people)

Sure- I don't mean to imply that it isn't. I can absolutely see how that's inexpensive for 100gbe equipment. That was more for the benefit of others like myself, who were wondering if "bargain" was comparative, or inexpensive enough that it might be worth buying one next time they upgraded switches. For me personally it's still an order of magnitude away from that.

https://mikrotik.com/product/crs305_1g_4s_in is the sweet spot right now for home users. Four 10g ports and a 1g, you can use the 1g for “uplink” to the internet and one of the 10g for your “big old Nortel gigabit switch with 10g uplink” and one for your Mac and two for your NAS and VM server. ;)

Direct cables are moderately cheap, and modules for 10g Ethernet aren’t insanely expensive.

Re: Ceph: A Journey to 1 TiB/s

#66

Post by 3abiton »

Palomides wrote: post #60
amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…

Trusting your maths, damn Apple did a great job on their M design.

Re: Ceph: A Journey to 1 TiB/s

#67

Post by amluto »

Palomides wrote: post #60
amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…

I think the Mac Mini has massively more compute than needed for this kind of work. It also has a power supply, and computer power supplies are generally not amazing at low output.

I’m imagining something quite specialized. Use a low frequency CPU with either vector units or even DMA engines optimized for the specific workloads needed, or go all out and arrange for data to be DMAed directly between the disk and the NIC.

Re: Ceph: A Journey to 1 TiB/s

#68

Post by Palomides »

amluto wrote: post #67
Palomides wrote: post #60

Earlier quoted context omitted.

here's a weird calculation: this cluster does something vaguely like 0.8 gigabits per second per watt (1 terabyte/s * 8 bits per byte * 1024 gb per tb / 34 nodes / 300 watts a new mac mini (super efficient arm system) runs around 10 watts in interactive usage and can do 10 gigabits per second network, so maybe 1 gigabit per second per watt of data so OP's cluster, back of the envelope, is basically the same bits per…

I think the Mac Mini has massively more compute than needed for this kind of work. It also has a power supply, and computer power supplies are generally not amazing at low output. I’m imagining something quite specialized. Use a low frequency CPU with either vector units or even DMA engines optimized for the specific workloads needed, or go all out and arrange for data to be DMAed directly between the disk and the NI…

sounds like a DPU (mellanox bluefield for example), they're entire ARM systems with a high speed NIC all on a PCIe card, I think the bluefield ones can even directly interface over the bus to nvme drives without the host system involved

Re: Ceph: A Journey to 1 TiB/s

#69

Post by KeplerBoy »

CyberDildonics wrote: post #59
KeplerBoy wrote: post #58

Earlier quoted context omitted.

I was talking about Gigabit/s, not Gigabyte/s. The article however actually talks about Terabyte/s scale, albeit not over a single node.

800 gigabits is 100 gigabytes which is still more than PCIe 5.0 16x 64 gigabyte per second bandwidth. You said there were 800 gigabit network cards, I'm wondering how that much bandwidth makes it to the card in the first place. The article however actually talks about Terabyte/s scale, albeit not over a single node. This does not have anything to do with what you originally said, you were talking about 800gb single p…

Yes, apparently I was mistaken about the NICs. They don't seem to be available yet.

But it's not a PCIe limitation. There are PCIe devices out there which use 32 lanes, so you could achieve the bandwidth even on PCIe5.

https://www.servethehome.com/ocp-nic-3-0-form-factors-quick-...

Re: Ceph: A Journey to 1 TiB/s

#70

Post by jseutter »

einpoklum wrote: post #53

Where can I read about the rationale for ceph as a project? I'm not familiar with it.

http://www.45drives.com/blog/ceph/what-is-ceph-why-our-custo... is a pretty good introduction. Basically you can take off-the-shelf hardware and keep expanding your storage cluster and ceph will scale fairly linearly up through hundreds of nodes. It is seeing quite a bit of use in things like Kubernetes and OpenShift as a cheap and cheerful alternative to SANs. It is not without complexity, so if you don't know you need it, it's probably not worth the hassle.
Post reply on HN