Ceph: A Journey to 1 TiB/s • Hacker News Board

Re: Ceph: A Journey to 1 TiB/s

#111

Post by sixdonuts » Sat, Jan 20, 2024, 4:57 AM UTC

sgarland wrote: post #61

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

Yes. I first tried it with Rook, and that was a disaster, so I shifted to Longhorn. That has had its own share of problems, and is quite slow. Finally, I let Proxmox manage Ceph for me, and it’s been a dream. So far I haven’t migrated my K8s workloads to it, but I’ve used it for RDBMS storage (DBs in VMs), and it works flawlessly. I don’t have an incredibly great setup, either: 3x Dell R620s (Ivy Bridge-era Xeons), a…

Just get a 25G switch and MM fiber. 25G switches are cheaper, use less power and can work with 10 and 25G SFPs.

Re: Ceph: A Journey to 1 TiB/s

#112

Post by amadio » Sat, Jan 20, 2024, 5:09 AM UTC

Nice article! We've also recently reached the mark of 1TB/s at CERN, but with EOS (https://cern.ch/eos), not ceph: https://www.home.cern/news/news/computing/exabyte-disk-stora...

Our EOS clusters have a lot more nodes, however, and use mostly HDDs. CERN also uses ceph extensively.

Re: Ceph: A Journey to 1 TiB/s

#113

Post by antongribok » Sat, Jan 20, 2024, 5:10 AM UTC

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

I run Ceph on some Raspberry Pi 4s. It's super reliable, and with cephadm it's very easy[1] to install and maintain.

My household is already 100% on Linux, so having a native network filesystem that I can just mount from any laptop is very handy.

Works great over Tailscale too, so I don't even have to be at home.

[1] I run a large install of Ceph at work, so "easy" might be a bit relative.

Re: Ceph: A Journey to 1 TiB/s

#114

Post by brobinson » Sat, Jan 20, 2024, 5:12 AM UTC

I'm curious what the performance difference would be on a modern kernel.

Re: Ceph: A Journey to 1 TiB/s

#115

Post by knicholes » Sat, Jan 20, 2024, 5:13 AM UTC

j33zusjuice wrote: post #84

knicholes wrote: post #46

Earlier quoted context omitted.

I had to run a bunch of benchmarks to compare speeds of not just AWS instance types, but actual individual instances in each type, as some NVME SSDs have been more used than others in order to lube up some Aerospike response times. Crazy.

Ad-tech, or?

Yeah. Serving profiles for customized ad selection.

Re: Ceph: A Journey to 1 TiB/s

#116

Post by amadio » Sat, Jan 20, 2024, 5:27 AM UTC

matheusmoreira wrote: post #72

nullwarp wrote: post #5

Earlier quoted context omitted.

There still are. As someone who has done both production and homelab deployments: unless you are specifically just looking for experience with it and just setting up a demo - don't bother. When it works, it works great - when it goes wrong it's a huge headache. Edit: As just an edit, if distributed storage is just something you are interested in there are much better options for a homelab setup: - seaweedfs has been…

I just want to hoard data. I hate having to delete stuff to make space. Things disappear from the web every day. I should hold onto them. My requirements for a storage solution are: > Single root file system > Storage device failure tolerance > Gradual expansion capability The problem with every storage solution I've ever seen is the lack of gradual expandability. I'm not a corporation, I'm just a guy. I don't have t…

EOS (https://cern.ch/eos, https://github.com/cern-eos/eos) is probably a bit more complicated than other solutions to setup and manage, but does allow to add/remove new disks and nodes serving data on the fly. This is essential to let us upgrade harware of the clusters serving experimental data with minimal to no downtime.

Re: Ceph: A Journey to 1 TiB/s

#117

Post by dcplaya » Sat, Jan 20, 2024, 5:30 AM UTC

antongribok wrote: post #113

matheusmoreira wrote: post #4

Does anyone have experience running ceph in a home lab? Last time I looked into it, there were quite significant hardware requirements.

I run Ceph on some Raspberry Pi 4s. It's super reliable, and with cephadm it's very easy[1] to install and maintain. My household is already 100% on Linux, so having a native network filesystem that I can just mount from any laptop is very handy. Works great over Tailscale too, so I don't even have to be at home. [1] I run a large install of Ceph at work, so "easy" might be a bit relative.

What are your speeds? Do you rub ceph FS too?

I'm trying to do similar.

Re: Ceph: A Journey to 1 TiB/s

#118

Post by aspenmayer » Sat, Jan 20, 2024, 5:38 AM UTC

ComputerGuru wrote: post #110

aspenmayer wrote: post #108

Earlier quoted context omitted.

Those numbers seem reasonable in that context. I first started using BitTorrent around that time as well, and it wasn't uncommon to see many users long-term seeding multiple hundreds of gigabytes of Linux ISOs alone. Here’s another usage scenario with data usage numbers I found a while back. > A 2004 paper published in ACM Transactions on Programming Languages and Systems shows how Hancock code can sift calling card…

I archived Hancock here over a decade ago, stumbled upon it via HN at the time if I’m not mistaken: https://github.com/mqudsi/hancock

That’s pretty cool. I remember someone on that repo from while back and was surprised to see their name pop up again. Thanks for archiving this!

Corinna Cortes et al wrote the paper(s) on Hancock and also the Communities of Interest paper referenced in the Wired article I linked to. She’s apparently a pretty big deal and went on to work at Google after her prestigious work at AT&T.

Hancock: A Language for Extracting Signatures from Data

https://scholar.google.com/citations?view_op=view_citation&h...

Hancock: A Language for Analyzing Transactional Data Streams

https://scholar.google.com/citations?view_op=view_citation&h...

Communities of Interest

https://scholar.google.com/citations?view_op=view_citation&h...

Re: Ceph: A Journey to 1 TiB/s

#119

Post by antongribok » Sat, Jan 20, 2024, 5:56 AM UTC

dcplaya wrote: post #117

antongribok wrote: post #113

Earlier quoted context omitted.

I run Ceph on some Raspberry Pi 4s. It's super reliable, and with cephadm it's very easy[1] to install and maintain. My household is already 100% on Linux, so having a native network filesystem that I can just mount from any laptop is very handy. Works great over Tailscale too, so I don't even have to be at home. [1] I run a large install of Ceph at work, so "easy" might be a bit relative.

What are your speeds? Do you rub ceph FS too? I'm trying to do similar.

It's been a while since I've done some benchmarks, but it can definitely do 40MB/s sustained writes, which is very good given the single 1GbE links on each node, and 5TB SMR drives.

Latency is hilariously terrible though. It's funny to open a text file over the network in vi, paste a long blob of text and watch it sync that line by line over the network.

If by "rub" you mean scrub, then yes, although I increased the scrub intervals. There's no need to scrub everything every week.

Re: Ceph: A Journey to 1 TiB/s

#120

Post by fiddlerwoaroof » Sat, Jan 20, 2024, 5:57 AM UTC

aspenmayer wrote: post #108

fiddlerwoaroof wrote: post #106

Earlier quoted context omitted.

It’s at least 20ish years ago: I remember an old sysadmin talking about managing petabytes before 2003

Those numbers seem reasonable in that context. I first started using BitTorrent around that time as well, and it wasn't uncommon to see many users long-term seeding multiple hundreds of gigabytes of Linux ISOs alone. Here’s another usage scenario with data usage numbers I found a while back. > A 2004 paper published in ACM Transactions on Programming Languages and Systems shows how Hancock code can sift calling card…

Yeah, at the other end of the scale, it sounds like Apple is now managing exabytes: https://read.engineerscodex.com/p/how-apple-built-icloud-to-...

This is pretty mind-boggling to me.