Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#121

Post by magicalhippo »

mrb wrote: post #40

I wanted to see how 1 TiB/s compares to the actual theoretical limits of the hardware. So here is what I found: The cluster has 68 nodes, each a Dell PowerEdge R6615 ( https://www.delltechnologies.com/asset/en-us/products/server... ). The R6615 configuration they run is the one with 10 U.2 drive bays. The U.2 link carries data over 4 PCIe gen4 lanes. Each PCIe lane is capable of 16 Gbit/s. The lanes have negligible ~…

They're benchmarking random IO though, and the disks can "only" do a bit over 1000k random 4k read IOPS, which translates to about 5 GiB/s. With 320 OSDs thats around 1.6 TiB/s.

At least thats the number I could find. Not exactly tons of reviews on these enterprise NVMe disks...

Still, that seems like a good match to the NICs. At this scale most workloads will likely appear as random IO at the storage layer anyway.

Re: Ceph: A Journey to 1 TiB/s

#122

Post by antongribok »

one_buggy_boi wrote: post #93

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I'd like to move to a cheaper cfs that can compete with systems like Oracle's clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven't seen much outside of it or ocfs that similarly scales with acceptable latency

Not sure about putting DBs on CephFS directly, but Ceph RBD can definitely run RDBMS workloads.

You need to pay attention to the kind of hardware you use, but you can definitely get Ceph down to 0.5-0.6 ms latency on block workloads doing single thread, single queue, sync 4K writes.

Source, I run Ceph at work doing pretty much this.

Re: Ceph: A Journey to 1 TiB/s

#123

Post by peter_d_sherman »

lmz wrote: post #98
wmf wrote: post #89

Earlier quoted context omitted.

what if I want a Linux ... that doesn't manage a local file system or local storage at all [but] operates solely using the network, solely using a distributed file system Linux can boot from NFS although that's kind of lost knowledge. Booting from CephFS might even be possible if you put the right parts in the initrd.

NFS root docs here https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs...

NFS is an excellent point!

NFS (now that I think about it!) -- brings up two additional software engineering considerations:

1) Distributed file system protocol.

2) Software that implements that distributed (or at least remote/network) file system -- via that file system protocol.

NFS is both.

That's not a bad thing(!) -- but ideally from a software engineering "separation of concerns" perspective, this future software layer/level would ideally be decoupled from the underlying protocol -- that is, it might have a "plug-in" protocol architecture, where various 3rd party file system protocols (somewhat analogous to drivers) could be "plugged-in"...

But NFS could definitely be used to boot/run Linux over the network, and is definitely a step in the right direction, and something worth evaluating for these purposes... its source code is definitely worth looking at...

So, an excellent point!

Re: Ceph: A Journey to 1 TiB/s

#124

Post by walrus01 »

amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

10 Gbps is increasingly obsolete with very low cost 100 Gbps switches and 100Gbps interfaces. Something would have to be really tiny and low cost to justify doing a ceph setup with 10Gbps interfaces now... If you're at that scale of very small stuff you are probably better off doing local NVME storage on each server instead.

Re: Ceph: A Journey to 1 TiB/s

#125

Post by antongribok »

epistasis wrote: post #62
alberth wrote: post #56

Ceph has an interesting history. It was created at Dreamhost (DH), for their internal needs by the founders. DH was doing effectively IaaS & PaaS before those were industry coined words (VPS, managed OS/database/app-servers). They spun Ceph off and Redhat bought it. https://en.wikipedia.org/wiki/DreamHost

A bit more to the story is that it was created also at UC Santa Cruz, by Sage Weil, a Dreamhost founder, while he was doing graduate work there. UCSC has had a lot of good storage research.

Sage is one of the nicest, down to earth, super smart individuals I've met.

I've talked to him at a few OpenStack and Ceph conferences, and he's always very patient answering questions.

Re: Ceph: A Journey to 1 TiB/s

#126

Post by redrove »

stuff4ben wrote: post #15

I used to love doing experiments like this. I was afforded that luxury as a tech lead back when I was at Cisco setting up Kubernetes on bare metal and getting to play with setting up GlusterFS and Ceph just to learn and see which was better. This was back in 2017/2018 if I recall. Good ole days. Loved this writeup!

A Heketi man! I had the same experience around the same years, what a blast. Everything was so new..and broken!

Re: Ceph: A Journey to 1 TiB/s

#127

Post by eurekin »

evanreichard wrote: post #90
amluto wrote: post #30

I wish someone would try to scale the nodes down. The system described here is ~300W/node for 10 disks/node, so 30W or so per disk. That’s a fair amount of overhead, and it also requires quite a lot of storage to get any redundancy at all. I bet some engineering effort could divide the whole thing by 10. Build a tiny SBC with 4 PCIe lanes for NVMe, 2x10GbE (as two SFP+ sockets), and a just-fast-enough ARM or RISC-V C…

I used to run a 5 node Ceph cluster on a bunch of ODROID-HC2's [0]. Was a royal pain to get installed (armhf processor). But once it was running it worked great. Just slow with the single 1Gb NIC. Was just a learning experience at the time. [0] https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/

Same here, but on PI 4b's. 6 node cluster with a 2tb hdd and 512 Tb ssd per node. CEPH made a huge impression on me, as in I didn't recognize how extensive the package was. I went up to 122mb/s and thought it's too little for my hack-NAS replacement :)

The functionality: mixing various pool types on the same set of SSD's, different redundancy types (erasure coded, replicated) was very impressive. Now I can't help but look down at a RAID NAS in comparision. Still, some extra packages like the NFS exporter were not ready for the arm architecture

Re: Ceph: A Journey to 1 TiB/s

#128

Post by kylegalbraith »

This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB/s and 3,000 IOPS to 900 MB/s and 30,000 IOPS.

The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something.

It’s been a massive improvement for our caching system.

[0] https://depot.dev/blog/cache-v2-faster-builds

Re: Ceph: A Journey to 1 TiB/s

#129

Post by theyinwhy »

amadio wrote: post #112

Nice article! We've also recently reached the mark of 1TB/s at CERN, but with EOS ( https://cern.ch/eos ), not ceph: https://www.home.cern/news/news/computing/exabyte-disk-stora... Our EOS clusters have a lot more nodes, however, and use mostly HDDs. CERN also uses ceph extensively.

Great! What's your take on ceph? Is the idea to migrate to EOS long term?

Re: Ceph: A Journey to 1 TiB/s

#130

Post by patrakov »

antongribok wrote: post #122
one_buggy_boi wrote: post #93

Is modern Ceph appropriate for transactional database storage, how is the IO latency? I'd like to move to a cheaper cfs that can compete with systems like Oracle's clustered file system or DBs backed by something like Veritas. Veritas supports multi-petabyte DBs and I haven't seen much outside of it or ocfs that similarly scales with acceptable latency

Not sure about putting DBs on CephFS directly, but Ceph RBD can definitely run RDBMS workloads. You need to pay attention to the kind of hardware you use, but you can definitely get Ceph down to 0.5-0.6 ms latency on block workloads doing single thread, single queue, sync 4K writes. Source, I run Ceph at work doing pretty much this.

It is important to specify which kind of latency percentile this is. Checking on a customer's cluster (made from 336 SATA SSDs in 15 servers, so not the best one in the world):

  50th percentile = 1.75 ms
  90th percentile = 3.15 ms
  99th percentile = 9.54 ms
That's with 700 MB/s of reads and 200 MB/s of writes, or approximately 7000 reads IOPS and 9000 writes IOPS.
Post reply on HN