Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

201–210 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#201

Post by amadio »

ComputerGuru wrote: post #185
amadio wrote: post #162

Earlier quoted context omitted.

EOS and ceph have different use cases at CERN. EOS holds physics data and user data in CERNBox, while ceph is used for a lot of the rest (e.g. storage for VMs, and other applications). So both will continue to be used as they are now. CERN has over 100PB on ceph.

Is there a reason you run both and don't converge on one or the other?

Yes, let me expand a bit on the answer above.

EOS was designed and developed with the unique needs of the LHC experiments in mind. The advantages it has are features used by the experiments, like support for remote access via the XRootD protocol, which is used for data analysis with ROOT (only the parts of files needed by an analysis are downloaded); rich support for client authentication methods (Kerberos, X509, OIDC, etc); and support to also FUSE mount everything to give a convenient POSIX-like view of the data. EOS needs to sustain ingestion of data at high rates from experiments (10s of GB/s each) for several months at a time during data taking without any downtime, while at the same time having tens of thousands of clients connected reading data as well. It's also integrated with the CERN Tape Archive (CTA) and File Tranfer Service (FTS), used for long term archival and data management across sites, respectively.

In the cases where block/object storage is needed, like storage for VMs, S3 storage for various uses, etc, then ceph is better suited. It has lower latency and EOS does not offer block-level access. In addition to providing storage services for OpenStack/Openshift, ceph is used to provide storage to back AFS and CVMFS, for example. CVMFS is another interesting piece of CERN's infrastructure, it's a read-only, HTTP-based FUSE filesystem used to distribute the software used by the experiments to grid sites around the world. Dan van der Ster, mentioned in the article above, has a good overview of ceph usage at CERN here: https://youtu.be/2I_U2p-trwI?si=Tsq4h8NIu4vSZQwt

If you are interested in EOS, we have the EOS workshop coming up in March: https://indico.cern.ch/event/1353101/

Re: Ceph: A Journey to 1 TiB/s

#202

Post by louwrentius »

yencabulator wrote: post #193
louwrentius wrote: post #146

Earlier quoted context omitted.

From my dated experience, Ceph is absolutely amazing but latency is indeed a relative weak spot. Everything has a trade-off and for Ceph you get a ton of capability but latency is such a trade-off. Databases - depending on requirements - may be better off on regular NVMe and not on Ceph.

It's pretty unfair to compare latency of a local NVMe SSD to over-the-network 3x replicated storage. "It's faster if I do less." [Disclaimer: ex-Inktank employee]

I don’t think it’s unfair, there are applications that still are ok with Ceph latencies: I bet it’s good enough for a ton of things.

But not all things.

Re: Ceph: A Journey to 1 TiB/s

#203

Post by dcplaya »

antongribok wrote: post #119
dcplaya wrote: post #117

Earlier quoted context omitted.

What are your speeds? Do you rub ceph FS too? I'm trying to do similar.

It's been a while since I've done some benchmarks, but it can definitely do 40MB/s sustained writes, which is very good given the single 1GbE links on each node, and 5TB SMR drives. Latency is hilariously terrible though. It's funny to open a text file over the network in vi, paste a long blob of text and watch it sync that line by line over the network. If by "rub" you mean scrub, then yes, although I increased the…

Rub, scrub... Was really supposed to be run but it got the answer I was looking for!

Right now, I'm doing about half of that but I am rcloning data between 2 ceph clusters. (Proxmox and a arm rook ceph cluster).

Re: Ceph: A Journey to 1 TiB/s

#204

Post by e12e »

kylegalbraith wrote: post #128

This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB/s and 3,000 IOPS to 900 MB/s and 30,000 IOPS. The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something. It’s been a massive improvement…

Did you host ebs on bare metal? How are you hosting ceph - your own/rented metal, ec2 - VMs?

Wasn't immediately clear to me from the blog.

Re: Ceph: A Journey to 1 TiB/s

#205

Post by e12e »

yencabulator wrote: post #193
louwrentius wrote: post #146

Earlier quoted context omitted.

From my dated experience, Ceph is absolutely amazing but latency is indeed a relative weak spot. Everything has a trade-off and for Ceph you get a ton of capability but latency is such a trade-off. Databases - depending on requirements - may be better off on regular NVMe and not on Ceph.

It's pretty unfair to compare latency of a local NVMe SSD to over-the-network 3x replicated storage. "It's faster if I do less." [Disclaimer: ex-Inktank employee]

No, it's important when planning - eg: one big database cluster that provides db-as-a-service (but maybe needs some dedicated ops resources) vs smaller DBs with virtualized storage on ceph (ops resources for ceph cluster and vm tools like k8s).

If the latter is too slow for your typical usage...

Re: Ceph: A Journey to 1 TiB/s

#206

Post by yencabulator »

e12e wrote: post #205
yencabulator wrote: post #193

Earlier quoted context omitted.

It's pretty unfair to compare latency of a local NVMe SSD to over-the-network 3x replicated storage. "It's faster if I do less." [Disclaimer: ex-Inktank employee]

No, it's important when planning - eg: one big database cluster that provides db-as-a-service (but maybe needs some dedicated ops resources) vs smaller DBs with virtualized storage on ceph (ops resources for ceph cluster and vm tools like k8s). If the latter is too slow for your typical usage...

Oh, don't get me wrong, you will pay a price for disaggregated highly available storage, and you might need to evaluate whether you want to pay that price or not. But those are two very different worlds, and only one of them gives you elastic disk size, replication, scale-out throughput, and so on.

GP makes Ceph sounds worse than it is, when reality is that just shoving all your reads & writes over the network, writes multiple times because of replication, is gonna cost you no matter what tech you build that with.

Re: Ceph: A Journey to 1 TiB/s

#208

Post by ComputerGuru »

amadio wrote: post #201
ComputerGuru wrote: post #185

Earlier quoted context omitted.

Is there a reason you run both and don't converge on one or the other?

Yes, let me expand a bit on the answer above. EOS was designed and developed with the unique needs of the LHC experiments in mind. The advantages it has are features used by the experiments, like support for remote access via the XRootD protocol, which is used for data analysis with ROOT (only the parts of files needed by an analysis are downloaded); rich support for client authentication methods (Kerberos, X509, OID…

Super cool! Thanks for taking the time to explain.

Re: Ceph: A Journey to 1 TiB/s

#209

Post by kylegalbraith »

e12e wrote: post #204
kylegalbraith wrote: post #128

This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB/s and 3,000 IOPS to 900 MB/s and 30,000 IOPS. The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something. It’s been a massive improvement…

Did you host ebs on bare metal? How are you hosting ceph - your own/rented metal, ec2 - VMs? Wasn't immediately clear to me from the blog.

We started with AWS EBS volumes with BuildKit on EC2. We've now moved to BuildKit on EC2 and a Ceph storage cluster on bare metal EC2 instances.

Re: Ceph: A Journey to 1 TiB/s

#210

Post by up2isomorphism »

PiratesScorn wrote: post #169
up2isomorphism wrote: post #166

This is an insanely expensive cluster built to show a benchmark. 68 node cluster serving only 15TB storage in total.

The purpose of the benchmarking was to validate the design of the cluster and to identify any issues before going into production, so it achieved exactly that objective. Without doing this work a lot of performance would have been left on the table before the cluster could even get out the door. As per the blog, the cluster is now in a 6+2 EC configuration for production which gives ~7PiB usable. Expensive yes, but w…

You are talking different thing. I don’t care what “purpose “ you want to achieve. I merely point out this performance number is mediocre at best, because the enormous computing power thrown at it , wether you like it or not.

To put it into perspective there are 68 nodes with 98 hard thread each, means only 1000/7000 = 140MB/s per thread or 280MB/s per core, and that’s not that impressive, to be honest.

Post reply on HN