Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cheyenne Super Computer Auction

gsaauctions.gov

Post reply on HN
Print view View original

41–50 of 159 posts

Re: Cheyenne Super Computer Auction

#41

Post by calaphos »

jeffbee wrote: post #7

I wonder what the point of liquid cooling such a system was. Were they pressed for space?

Has been really common in HPC for quite a while. I presume the higher interconnect/network of hpc favour the higher density of liquid cooling. Hardware utilization is also higher compared to normal datacenters, so the additional efficiency vs air cooling is more useful.

Re: Cheyenne Super Computer Auction

#42

Post by observationist »

mk_stjames wrote: post #37
observationist wrote: post #29

Earlier quoted context omitted.

In comparison, 18 A100 GPUs would have 5.6 petaflops and 1.4 TB vram, consuming 5.6 kw. The speed of processing and interconnect is orders of magnitude faster for an A100 cluster - 1 8 gpu pod server will cost around $200k, so around $600k more or less beats the supercomputer performance (price I'm searching seems wildly variable, please correct me if I'm wrong.)

The Cheyenne numbers are 5.34 petaflops of *FP64*. The 5.6PF you quote for 18 A100's would be in BF16. Not comparable. The A100 can only do 9.746 TFLOPS in FP64. So you would need 548 A100's to match the FP64 performance of the Cheyenne.

Thanks, glad you guys caught that - could be generous and allow the tensor core tflops, since you'd more than likely be using a100 pods for something cuda optimized, in which case 19.5 tflops fp64 at peak per GPU, roughly 267 would be needed, or 34 pods, at $6.8 million, with 21.76 TB vram and 81 kw power consumption.

Double those for raw fp64.

Re: Cheyenne Super Computer Auction

#45

Post by convolvatron »

jeffbee wrote: post #7

I wonder what the point of liquid cooling such a system was. Were they pressed for space?

for large machines the air setup is really less efficient and takes up alot of space. you end up building a big room with a pressurized floor which is completely ringed by large ac units. you have to move alot of air through the floor bringing it up through the cabinets and back through to the acs. its also a big control systems problem, you need to get the air through the cabinets evenly, so you need variable speed fans or controlled ducts..and those need to be adaptive but not oscillate.

with a water cooled setup you can move alot more heat through your pipes just be increasing flow rate. so you need pumps instead of fans. and now your machine room isn't a mini-hurricane, and you can more flexibly deal with the waste heat.

Re: Cheyenne Super Computer Auction

#46

Post by vel0city »

sambull wrote: post #11
h2odragon wrote: post #9

Earlier quoted context omitted.

Recyclers.

any I've dealt with definitely wouldn't touch the 'you need to hire professional movers costing you $10k's of dollars to get it out of the facility' stipulation - they seem to prefer the 'where's the location of the storage shed' situation.

There's 8,064 E5-2697v4's in this. Those go on ebay for ~$50/ea. That's $400,000 of just CPUs to sell.

If the winning bid is $100k, you spend $40k to move it out of there, another $10k warehousing it while selling everything on ebay, and you're still up $250k on the processors alone.

Re: Cheyenne Super Computer Auction

#48

Post by EvanAnderson »

vel0city wrote: post #46
sambull wrote: post #11

Earlier quoted context omitted.

any I've dealt with definitely wouldn't touch the 'you need to hire professional movers costing you $10k's of dollars to get it out of the facility' stipulation - they seem to prefer the 'where's the location of the storage shed' situation.

There's 8,064 E5-2697v4's in this. Those go on ebay for ~$50/ea. That's $400,000 of just CPUs to sell. If the winning bid is $100k, you spend $40k to move it out of there, another $10k warehousing it while selling everything on ebay, and you're still up $250k on the processors alone.

> That's $400,000 of just CPUs to sell.

But do you crater the market for those CPUs? What's the demand for 2016-era Xeons and how much of their price comes down to supply?

Re: Cheyenne Super Computer Auction

#49

Post by unnouinceput »

Isamu wrote: post #4

>Components of the Cheyenne Supercomputer Installed Configuration: SGI ICE™ XA. E-Cells: 14 units weighing 1500 lbs. each. E-Racks: 28 units, all water-cooled Nodes: 4,032 dual socket units configured as quad-node blades Processors: 8,064 units of E5-2697v4 (18-core, 2.3 GHz base frequency, Turbo up to 3.6GHz, 145W TDP) Total Cores: 145,152 Memory: DDR4-2400 ECC single-rank, 64 GB per node, with 3 High Memory E-Cells…

>...totaling 313,344 GB

Can you imagine the RAMDisk? Yes, you can. Especially in 20 years when it will be the norm. And also the Windows version that will require half of it in order to run /s

Post reply on HN