Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ceph: A Journey to 1 TiB/s

ceph.io

Post reply on HN
Print view View original

211–216 of 216 posts

Re: Ceph: A Journey to 1 TiB/s

#211

Post by mrunkel »

up2isomorphism wrote: post #210
PiratesScorn wrote: post #169

Earlier quoted context omitted.

The purpose of the benchmarking was to validate the design of the cluster and to identify any issues before going into production, so it achieved exactly that objective. Without doing this work a lot of performance would have been left on the table before the cluster could even get out the door. As per the blog, the cluster is now in a 6+2 EC configuration for production which gives ~7PiB usable. Expensive yes, but w…

You are talking different thing. I don’t care what “purpose “ you want to achieve. I merely point out this performance number is mediocre at best, because the enormous computing power thrown at it , wether you like it or not. To put it into perspective there are 68 nodes with 98 hard thread each, means only 1000/7000 = 140MB/s per thread or 280MB/s per core, and that’s not that impressive, to be honest.

> This is an insanely expensive cluster built to show a benchmark. 68 node cluster serving only 15TB storage in total.

This reads to me (and the OP) that you are saying the purpose of this "insanely expensive cluster" was to "show a benchmark."

That's what OP is addressing in his response. No where do you mention anything about performance.

Re: Ceph: A Journey to 1 TiB/s

#212

Post by kaliszad »

What surprises me is, why they went with the harder to cool 1U nodes and 10 SSDs/2x100Gb NICs instead of 2U nodes with 24 SSDs/2x200 or even 400Gb NICs. They could remove the network bottleneck and save on power thanks to larger, lower speed fans and less CPU packages, possibly with more cores per socket though. Also, having a smaller number of nodes increases the blast radius but with even 34 nodes this is probably not such a problem. However, with less nodes they could have a flatter network with 4 switches or so too.

Re: Ceph: A Journey to 1 TiB/s

#213

Post by up2isomorphism »

mrunkel wrote: post #211
up2isomorphism wrote: post #210

Earlier quoted context omitted.

You are talking different thing. I don’t care what “purpose “ you want to achieve. I merely point out this performance number is mediocre at best, because the enormous computing power thrown at it , wether you like it or not. To put it into perspective there are 68 nodes with 98 hard thread each, means only 1000/7000 = 140MB/s per thread or 280MB/s per core, and that’s not that impressive, to be honest.

> This is an insanely expensive cluster built to show a benchmark. 68 node cluster serving only 15TB storage in total. This reads to me (and the OP) that you are saying the purpose of this "insanely expensive cluster" was to "show a benchmark." That's what OP is addressing in his response. No where do you mention anything about performance.

1TB/second is a benchmark number, and obviously it is trying to impress. This is already a purpose without further clarification, on the other hand it does not necessarily mean it can not have other purpose - which again looks an extremely expensive cluster for that purpose. And I do not see the reason to down vote except some one got hurt in the feeling with a fact. With the actually configuration shown, it is just not that performant nor economical, as I said in the reply, if you had read anything about it.

Re: Ceph: A Journey to 1 TiB/s

#214

Post by hirako2000 »

amluto wrote: post #178
hirako2000 wrote: post #174

Earlier quoted context omitted.

I checked selling prices of those racks + top end SSDs, this 1Tb/s achievement runs on $4 million worth of hardware cluster. Or more I didn't check the networking interface costs. But yeah could run on commodity hardware. Not sure those highly efficient arm packaged for a premium from Apple would beat the Dell racks though regarding throughput relative to hardware investment costs.

Dell’s list prices have essentially nothing to do with the prices that any competent buyer would actually pay, especially when storage is involved. Look at the prices of Dell disks, which are nothing special compared to name brand disks of equal or better spec and much lower list price. I don’t know what discount large buyers get, but I wouldn’t be surprised if it’s around 75%.

Agreed and the specs in the story in fact show they didn't provision add-ons such as specific SSDs from dell.

Still well over $1M for the cluster.. skeletons of racks with just CPUs and ram.

Re: Ceph: A Journey to 1 TiB/s

#215

Post by PiratesScorn »

kaliszad wrote: post #212

What surprises me is, why they went with the harder to cool 1U nodes and 10 SSDs/2x100Gb NICs instead of 2U nodes with 24 SSDs/2x200 or even 400Gb NICs. They could remove the network bottleneck and save on power thanks to larger, lower speed fans and less CPU packages, possibly with more cores per socket though. Also, having a smaller number of nodes increases the blast radius but with even 34 nodes this is probably…

Blast radius is the primary factor as you say and just generally makes things like patching and HW replacements less stressful. The racks and switches already exist and are heavily utilised for other purposes so the additional physical footprint for ceph is pretty tiny :)

Re: Ceph: A Journey to 1 TiB/s

#216

Post by markhpc »

up2isomorphism wrote: post #210
PiratesScorn wrote: post #169

Earlier quoted context omitted.

The purpose of the benchmarking was to validate the design of the cluster and to identify any issues before going into production, so it achieved exactly that objective. Without doing this work a lot of performance would have been left on the table before the cluster could even get out the door. As per the blog, the cluster is now in a 6+2 EC configuration for production which gives ~7PiB usable. Expensive yes, but w…

You are talking different thing. I don’t care what “purpose “ you want to achieve. I merely point out this performance number is mediocre at best, because the enormous computing power thrown at it , wether you like it or not. To put it into perspective there are 68 nodes with 98 hard thread each, means only 1000/7000 = 140MB/s per thread or 280MB/s per core, and that’s not that impressive, to be honest.

Hi, Author here.

Large reads tend to require the least CPU of all of the tests that we ran in the post. This is especially true in a 3X replication scenario where reads are serviced by a single OSD like in the 1 TiB/s test. CPU is far more important for small random writes, and also can be important when using erasure coding and/or msgr level encryption.

So the premise that you can only achieve 280MB/s per core is misleading. This cluster wasn't bottlenecked by the CPUs for large reads. Having said that, CPU makes up only a small portion of the overall cost for an NVMe deployment like this. Investing a relatively small amount of money to achieve a higher core to nvme ratio provides a better balance across all workloads and more flexibility when enabling features that consume additional CPU.

Post reply on HN