Ceph: A Journey to 1 TiB/s • Hacker News Board

Re: Ceph: A Journey to 1 TiB/s

#91

Post by mrb » Sat, Jan 20, 2024, 1:23 AM UTC

wmf wrote: post #87

mrb wrote: post #40

I wanted to see how 1 TiB/s compares to the actual theoretical limits of the hardware. So here is what I found: The cluster has 68 nodes, each a Dell PowerEdge R6615 ( https://www.delltechnologies.com/asset/en-us/products/server... ). The R6615 configuration they run is the one with 10 U.2 drive bays. The U.2 link carries data over 4 PCIe gen4 lanes. Each PCIe lane is capable of 16 Gbit/s. The lanes have negligible ~…

I think PCIe TLP overhead and NVMe commands account for the difference between 7 and 8 GB/s.

You are probably right. Reading some old notes of mine when I was fine-tuning PCIe bandwith on my ZFS server, I had discovered back then that a PCIe Max_Payload_Size of 256 bytes limited usable bandwidth to about 74% of the link's theoretical max. I had calculated that 512 and 1024 bytes (the maximum) would raise it to respectively about 86% and 93% (but my SATA controllers didn't support a value greater than 256.)

Re: Ceph: A Journey to 1 TiB/s

#92

Post by chx » Sat, Jan 20, 2024, 1:26 AM UTC

There was a point in history when the total amount of digital data stored worldwide reached 1TiB for the first time. It is extremely likely this day was within the last sixty years.

And here we are moving that amount of data every second on the servers of a fairly random entity. We not talking of a nation state or a supranatural research effort.

Re: Ceph: A Journey to 1 TiB/s

#93

Post by one_buggy_boi » Sat, Jan 20, 2024, 1:30 AM UTC

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I'd like to move to a cheaper cfs that can compete with systems like Oracle's clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven't seen much outside of it or ocfs that similarly scales with acceptable latency

Re: Ceph: A Journey to 1 TiB/s

#94

Post by samcat116 » Sat, Jan 20, 2024, 1:48 AM UTC

one_buggy_boi wrote: post #93

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I'd like to move to a cheaper cfs that can compete with systems like Oracle's clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven't seen much outside of it or ocfs that similarly scales with acceptable latency

Latency is quite poor, I wouldn't recommend running high performance database loads there.

Re: Ceph: A Journey to 1 TiB/s

#95

Post by samcat116 » Sat, Jan 20, 2024, 1:55 AM UTC

willglynn wrote: post #37

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

The hardware minimums are real, and the complexity floor is significant. Do not deploy Ceph unless you mean it. I started considering alternatives when my NAS crossed 100 TB of HDDs, and when a scary scrub prompted me to replace all the HDDs, I finally pulled the trigger. (ZFS resilvered everything fine, but replacing every disk sequentially gave me a lot of time to think.) Today I have far more HDD capacity and a fe…

Can I ask what you use all that storage for on your NAS?

Re: Ceph: A Journey to 1 TiB/s

#96

Post by roygbiv2 » Sat, Jan 20, 2024, 1:55 AM UTC

chromatin wrote: post #79

matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

> ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

This is also exactly how mine is done. Started off with a bunch of 2TB disks. I've now got a mixture of 16TB down to 4TB, all in the original pool.

Re: Ceph: A Journey to 1 TiB/s

#97

Post by geerlingguy » Sat, Jan 20, 2024, 2:34 AM UTC

asadhaider wrote: post #50

nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I thought it was popular for people running Proxmox clusters

It is, and if you have a few nodes with at least 10 GbE networking, it's certainly the best clustered storage option I can think of.

Re: Ceph: A Journey to 1 TiB/s

#98

Post by lmz » Sat, Jan 20, 2024, 2:39 AM UTC

wmf wrote: post #89

peter_d_sherman wrote: post #83

Ceph is interesting... open source software whose only purpose is to implement a distributed file system... Functionally, Linux implements a file system (well, several!) as well (in addition to many other OS features) -- but (usually!) only on top of local hardware. There seems to be some missing software here -- if we examine these two paradigms side-by-side. For example, what if I want a Linux (or more broadly, a g…

what if I want a Linux ... that doesn't manage a local file system or local storage at all [but] operates solely using the network, solely using a distributed file system Linux can boot from NFS although that's kind of lost knowledge. Booting from CephFS might even be possible if you put the right parts in the initrd.

NFS root docs here https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs...

Re: Ceph: A Journey to 1 TiB/s

#99

Post by rafaelturk » Sat, Jan 20, 2024, 2:45 AM UTC

I'm playing a lot with MicroCeph. Its aopinionated low TOS, friendly setup of Ceph. Looking forward additional comments. Planning to use it in production and replace lots of NAS servers.

Re: Ceph: A Journey to 1 TiB/s

#100

Post by mikecoles » Sat, Jan 20, 2024, 2:47 AM UTC

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Works great, depending on what you want to do. Running on SBCs or computers with cheap sata cards will greatly reduce the performance. It's been running well for years after I found out the issues regarding SMR drives and the SATA card bottlenecks.

45Drives has a homelab setup if you're looking for a canned solution.