Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cheyenne Super Computer Auction

gsaauctions.gov

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Cheyenne Super Computer Auction

#91

Post by toast0 »

ansible wrote: post #65
EvanAnderson wrote: post #48

Earlier quoted context omitted.

> That's $400,000 of just CPUs to sell. But do you crater the market for those CPUs? What's the demand for 2016-era Xeons and how much of their price comes down to supply?

I presume no one is building new motherboards for those processors either. While there is old stock laying around, you really need to run those systems close to as-is for them to be useful.

These are high spec cpus for the socket though. Lots of room for people with compatible boards that want to upgrade.

There's a lot of low budget hosting with old Xeon systems (I'm paying $30/month for a dual westmere system; but I've seen plenty of offers on newer gear); you can still do a lot with an 18 core Broadwell, if the power is cheap.

Re: Cheyenne Super Computer Auction

#92

Post by latchkey »

mk_stjames wrote: post #89
latchkey wrote: post #63

Earlier quoted context omitted.

AMD MI300x is 163.4 TFLOPS in FP64. 33 of them, which would also have 6,336TB of memory. I'll have way more than that in my next purchase order. It is really fun to build a super computer.

I'm an amateur, but I have code that I think could probably dispatch threads pretty efficiently on the Cheyenne thru it's management system simply because it's all xeons distributed. If I can run it on my personal 80-core cluster, I could get it to run on Cheyenne back then. But hitting the roofline on those AMD GPGPU's? I'd probably get nowhere fucking close. That is the thing that Cheyenne was built for. People doi…

It is wild how much compute has grown.

I put dual Epyc 9754 into my first box of MI300x.

That's 256 cores + 8x MI300x, in a single box.

Agreed, it is a great solution for CFD, which is definitely one workload I'd love to host.

Re: Cheyenne Super Computer Auction

#93

Post by toast0 »

michaelt wrote: post #62
jeffbee wrote: post #7

I wonder what the point of liquid cooling such a system was. Were they pressed for space?

The heat's going to be leaving the building in liquid-filled pipes, however you architect it. And with 1.7MW of peak power consumption, a nontrivial amount of liquid. It's just a question of whether you want to add air and refrigerant into the mix. It seems they're decommissioning it partly due to "faulty quick disconnects causing water spray" though, so an air cooling stage would have had its benefits...

> The heat's going to be leaving the building in liquid-filled pipes

In the right climate, and the right power density, you can use outside air for cooling, at least part of the time. Unlikely at this scale of machine, but there was a lot of work towards datacenter siting in the 2010s to find places were ambient cooling would significantly reduce the power needed to operate.

Re: Cheyenne Super Computer Auction

#94

Post by mchannon »

I once bought a far larger supercomputer. It was 1/8 (roughly) of ASCI Blue Mountain. 72 racks. Commissioned in 1998 as #1 or #2 on the TOP500, officially decommissioned in 2004, purchased my 1/8 for $7k in ~2005.

Moving 72 racks was NOT easy. After paying substantial storage fees, I rented a 1500sf warehouse after selling off a few of them and they filled it up. Took a while to get 220V/30A service in there to run just one of them for testing purposes. Installing IRIX was 10x worse than any other OS. Imagine 8 CD's and you had to put them each in 2x during the process. Luckily somebody listed a set on eBay. SGI was either already defunct or just very unfriendly to second hand owners like myself.

The racks ran SGI Origin 2000s with CRAYlink interlinks. Sold 'em off 1-8 at a time, mainly to render farms. Toy Story had been made on similar hardware. The original NFL broadcasts with that magic yellow first down line were synthesized with similar hardware. One customer did the opening credits for a movie with one of my units.

I remember still having half of them around when Bitcoin first came out. It never occurred to me to try to mine with them, though I suspect if I'd been able to provide sufficient electrical service for the remainder, Satoshi and I would've been neck-and-neck for number of bitcoins in our respective wallets.

The whole exercise was probably worthwhile. I learned a lot, even if it does feel like seven lifetimes ago.

Re: Cheyenne Super Computer Auction

#95

Post by toast0 »

organsnyder wrote: post #58
humansareok1 wrote: post #54

Earlier quoted context omitted.

Auctions by default almost always undercut the actual market value so no not really?

Isn't the winning bid the actual market value, by definition?

Depends on the terms of the auction. If we take the California legal definition of Fair Market Value for real estate:

> The fair market value of the property taken is the highest price on the date of valuation that would be agreed to by a seller, being willing to sell but under no particular or urgent necessity for so doing, nor obliged to sell, and a buyer, being ready, willing, and able to buy but under no particular necessity for so doing, each dealing with the other with full knowledge of all the uses and purposes for which the property is reasonably adaptable and available.

A 7 day auction on a complex product like this may be a little short to qualify with the necessity clauses, IMHO; there's a bit too much time pressure, and not enough time for a buyer to inspect and research.

Re: Cheyenne Super Computer Auction

#96

Post by h2odragon »

Tempest1981 wrote: post #67
h2odragon wrote: post #8

> the system is currently experiencing maintenance limitations due to faulty quick disconnects causing water spray. Given the expense and downtime associated with rectifying this issue in the last six months of operation, it's deemed more detrimental than the anticipated failure rate of compute nodes. Even the RAM has aged out... Very hard to justify running any of this; newer kit pays for itself in reduced power and…

Are you trying to discourage others from bidding, so you can swoop in and win the auction?

nah, i already have far more junk computers than i need.

I lusted after a Cray T3E once that I coulda had for $1k and trucking it across TN and NC; but even then I couldn't have run it. I'm two miles away from 3 phase power and even then couldn't have justified the power budget. At the time a slightly less scrap UltraSPARC 6k beat it up on running costs even with higher initial costs so i went with that instead. I did find a bunch of Alphas to do the byte swizzling tho. Ex "Titanic" render farm nodes.

I've been away from needing low budget big compute for a while, but having spent a few years watching the space i still can't help but go "ooo neat" and wonder what i could do with it.

https://en.wikipedia.org/wiki/Cray_T3E

Re: Cheyenne Super Computer Auction

#98

Post by [deleted] »

Isamu wrote: post #4

>Components of the Cheyenne Supercomputer Installed Configuration: SGI ICE™ XA. E-Cells: 14 units weighing 1500 lbs. each. E-Racks: 28 units, all water-cooled Nodes: 4,032 dual socket units configured as quad-node blades Processors: 8,064 units of E5-2697v4 (18-core, 2.3 GHz base frequency, Turbo up to 3.6GHz, 145W TDP) Total Cores: 145,152 Memory: DDR4-2400 ECC single-rank, 64 GB per node, with 3 High Memory E-Cells…

[deleted]

Re: Cheyenne Super Computer Auction

#99

Post by nickpsecurity »

mk_stjames wrote: post #37
observationist wrote: post #29

Earlier quoted context omitted.

In comparison, 18 A100 GPUs would have 5.6 petaflops and 1.4 TB vram, consuming 5.6 kw. The speed of processing and interconnect is orders of magnitude faster for an A100 cluster - 1 8 gpu pod server will cost around $200k, so around $600k more or less beats the supercomputer performance (price I'm searching seems wildly variable, please correct me if I'm wrong.)

The Cheyenne numbers are 5.34 petaflops of *FP64*. The 5.6PF you quote for 18 A100's would be in BF16. Not comparable. The A100 can only do 9.746 TFLOPS in FP64. So you would need 548 A100's to match the FP64 performance of the Cheyenne.

Also, supercomputers usually use general-purpose nodes supported by many standard tools, multiple methods of parallelization, and (for open standards) maybe multi-vendor. I imagine this one is much more flexible than A100’s.

Re: Cheyenne Super Computer Auction

#100

Post by nickpsecurity »

formerly_proven wrote: post #66
nickpsecurity wrote: post #55

My favorite part of SGI computers, like Altix and UV lines, was the NUMA memory with flexible interconnect. NUMA let you program a pile of CPU’s more like a single-node, multithreaded system. Then, the flexibility let you plug in CPU’s, graphics cards, or FPGA’s. That’s right into the low-latency, high-speed, memory bus. There was a company that made a card that connected AMD servers like that. I don’t know if such t…

Cray XT/SeaStar? iirc the interconnect ASIC pretends to be another peer CPU connected via HyperTransport. HPE Flex is similar, but works via QPI/UPI for Intel CPUs.

It was NUMAscale. It’s mentioned in this article with some others for comparison:

https://www.nextplatform.com/2015/07/16/what-if-numa-scaling...

Post reply on HN