Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

81–90 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#81

Post by NavinF »

CyberDildonics wrote: post #59
KeplerBoy wrote: post #58

Earlier quoted context omitted.

I was talking about Gigabit/s, not Gigabyte/s. The article however actually talks about Terabyte/s scale, albeit not over a single node.

800 gigabits is 100 gigabytes which is still more than PCIe 5.0 16x 64 gigabyte per second bandwidth. You said there were 800 gigabit network cards, I'm wondering how that much bandwidth makes it to the card in the first place. The article however actually talks about Terabyte/s scale, albeit not over a single node. This does not have anything to do with what you originally said, you were talking about 800gb single p…

I'm not aware of any 800G cards, but FYI a single Mellanox card can use two PCIe x16 slots to avoid NUMA issues on dual-socket servers: https://www.nvidia.com/en-us/networking/ethernet/socket-dire...

So the software infra for using multiple slots already exists and doesn't require any special config. Oh and some cards can use PCIe slots across multiple hosts. No idea why you'd want to do that, but you can.

Re: Ceph: A Journey to 1 TiB/s

#82

Post by matheusmoreira »

loeg wrote: post #6
matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Why would you bother with a distributed filesystem when you don't have to?

I'm indifferent towards the distributed nature thing. What I want is ceph's ability to pool any combination of drives of any make, model and capacity into organized redundant fault tolerant storage, and its ability to add arbitrary drives to that pool at any point in the system's lifetime. RAID-like solutions require identical drives and can't be easily expanded.

Re: Ceph: A Journey to 1 TiB/s

#83

Post by peter_d_sherman »

Ceph is interesting... open source software whose only purpose is to implement a distributed file system...

Functionally, Linux implements a file system (well, several!) as well (in addition to many other OS features) -- but (usually!) only on top of local hardware.

There seems to be some missing software here -- if we examine these two paradigms side-by-side.

For example, what if I want a Linux (or more broadly, a general OS) -- but one that doesn't manage a local file system or local storage at all?

One that operates solely using the network, solely using a distributed file system that Ceph, or software like Ceph, would provide?

Conversely, what if I don't want to run a full OS on a network machine, a network node that manages its own local storage?

The only thing I can think of to solve those types of problems -- is:

What if the Linux filesystem was written such that it was a completely separate piece of software, and a distributed file system like Ceph, and not dependent on the other kernel source code (although, still complilable into the kernel as most linux components normally are)...

A lot of work? Probably!

But there seems to be some software need for something between a solely distributed file system as Ceph is, and a completely monolithic "everything baked in" (but not distributed!) OS/kernel as Linux is...

Note that I am just thinking aloud here -- I probably am wrong and/or misinformed on one or more fronts!

So, kindly take this random "thinking aloud" post -- with the proverbial "grain of salt!" :-)

Re: Ceph: A Journey to 1 TiB/s

#84

Post by j33zusjuice »

knicholes wrote: post #46
stuff4ben wrote: post #15

I used to love doing experiments like this. I was afforded that luxury as a tech lead back when I was at Cisco setting up Kubernetes on bare metal and getting to play with setting up GlusterFS and Ceph just to learn and see which was better. This was back in 2017/2018 if I recall. Good ole days. Loved this writeup!

I had to run a bunch of benchmarks to compare speeds of not just AWS instance types, but actual individual instances in each type, as some NVME SSDs have been more used than others in order to lube up some Aerospike response times. Crazy.

Ad-tech, or?

Re: Ceph: A Journey to 1 TiB/s

#85

Post by AdamJacobMuller »

epistasis wrote: post #62
alberth wrote: post #56

Ceph has an interesting history. It was created at Dreamhost (DH), for their internal needs by the founders. DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers). They spun Ceph off and Redhat bought it. https://en.wikipedia.org/wiki/DreamHost

A bit more to the story is that it was created also at UC Santa Cruz, by Sage Weil, a Dreamhost founder, while he was doing graduate work there. UCSC has had a lot of good storage research.

I remember the first time I deployed ceph, would have been around 2010 or 2011, had some really major issues which would nearly resulted in data loss and due to someone else not realizing what "this cluster is experimental, do not store any important data here" meant, the data on ceph was the only copy of the irreplaceable data in the world, loosing the data would have been fairly catastrophic for us.

I ended up on the ceph IRC channel and eventually had Sage helping me fix the issues directly, helping me find bugs and writing patches to fix them in realtime.

Super amazingly nice guy that he was willing to help, never once chastised me for being so stupid (even though I was), also wicked smart.

Re: Ceph: A Journey to 1 TiB/s

#86

Post by bityard »

matheusmoreira wrote: post #72
nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

On a single host, you could do this with LVM. Add a pair of disks, make them a RAID 1, create a physical volume on them, then a volume group, then a logical volume with XFS on top. To expand, you add a pair of disks, RAID 1 them, and add them to the LVM. It's a little stupid, but it would work.

If multiple nodes are not off the table, also look into seaweedfs.

Also consider how (or if) you are going to back up your hoard of data.

Re: Ceph: A Journey to 1 TiB/s

#87

Post by wmf »

mrb wrote: post #40

I wanted to see how 1 TiB/s compares to the actual theoretical limits of the hardware. So here is what I found: The cluster has 68 nodes, each a Dell PowerEdge R6615 ( https://www.delltechnologies.com/asset/en-us/products/server... ). The R6615 configuration they run is the one with 10 U.2 drive bays. The U.2 link carries data over 4 PCIe gen4 lanes. Each PCIe lane is capable of 16 Gbit/s. The lanes have negligible ~…

I think PCIe TLP overhead and NVMe commands account for the difference between 7 and 8 GB/s.

Re: Ceph: A Journey to 1 TiB/s

#88

Post by matheusmoreira »

bityard wrote: post #86
matheusmoreira wrote: post #72

Earlier quoted context omitted.

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

On a single host, you could do this with LVM. Add a pair of disks, make them a RAID 1, create a physical volume on them, then a volume group, then a logical volume with XFS on top. To expand, you add a pair of disks, RAID 1 them, and add them to the LVM. It's a little stupid, but it would work. If multiple nodes are not off the table, also look into seaweedfs. Also consider how (or if) you are going to back up your h…

> Also consider how (or if) you are going to back up your hoard of data.

I actually emailed backblaze years ago about their supposedly unlimited consumer backup plan. Asked them if they would really allow me to dump into their systems dozens of terabytes of encrypted undeduplicable data. They responded that yes, they would. Still didn't believe them, these corporations never really mean it when they say unlimited. Plus they had no Linux software.

Re: Ceph: A Journey to 1 TiB/s

#89

Post by wmf »

peter_d_sherman wrote: post #83

Ceph is interesting... open source software whose only purpose is to implement a distributed file system... Functionally, Linux implements a file system (well, several!) as well (in addition to many other OS features) -- but (usually!) only on top of local hardware. There seems to be some missing software here -- if we examine these two paradigms side-by-side. For example, what if I want a Linux (or more broadly, a g…

what if I want a Linux ... that doesn't manage a local file system or local storage at all [but] operates solely using the network, solely using a distributed file system

Linux can boot from NFS although that's kind of lost knowledge. Booting from CephFS might even be possible if you put the right parts in the initrd.

Re: Ceph: A Journey to 1 TiB/s

#90

Post by evanreichard »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

I used to run a 5 node Ceph cluster on a bunch of ODROID-HC2's [0]. Was a royal pain to get installed (armhf processor). But once it was running it worked great. Just slow with the single 1Gb NIC.

Was just a learning experience at the time.

[0] https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/

Post reply on HN