Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cheyenne Super Computer Auction

gsaauctions.gov

Post reply on HN
Print view View original

121–130 of 159 posts

Re: Cheyenne Super Computer Auction

#121

Post by jasonwatkinspdx »

dekhn wrote: post #116
jasonwatkinspdx wrote: post #103

Earlier quoted context omitted.

At the ill fated Portland TechShop I took woodworking classes from a retired gentleman, who professionally was a pattern maker for molding cast metal parts. This made his approach to woodworking really interesting. He had a huge array of freestanding sander machines, including a disc sander with more than a yard diameter. For anyone unfamiliar, pattern makers would make wooden model versions of parts that were to be…

Does 1/64" precision really mean anything in wood, where small fluctuations in air moisture can cause > 1/64" distortion? I guess it's OK if you stay within a climate controlled area.

So he would build parts by first making an oversize rough blank of bonded layers of marine grade plywood in a big press. Then he'd rough cut it various ways on a big band saw. Then he'd work his way through using all the sanders to slowly approach the net shape. He used precision squares to measure bigger stuff and calipers for smaller stuff.

I can't tell you the exact stability of marine grade plywood, but I know it's about as good as you can get for a wooden material, and I doubt he'd go to the effort of such precise measurements if it didn't matter.

Re: Cheyenne Super Computer Auction

#122

Post by dylan604 »

jazzyjackson wrote: post #69
michaelt wrote: post #57

Earlier quoted context omitted.

So getting 8,064 of them for $3,085 - 38 cents per CPU - is great value for money!

this is basically "free grand piano" - not so free once you hire the movers and tuners

At least a piano doesn't require power and cooling to operate.

Re: Cheyenne Super Computer Auction

#123

Post by dekhn »

latchkey wrote: post #120
dekhn wrote: post #115

Earlier quoted context omitted.

I used to build small clusters and use supercomputers and I can't imagine it's fun to build a super computer. It requires a massive infrastructure and significant employee base, and individual component failures can take down entire jobs. Finding enough jobs to keep the system loaded 24/7 while also keeping the interconnect (which was 15-20% of the total system cost) busy, and finding the folks who can write such job…

Thanks for the feedback. You make a lot of good points. I've built a 150,000 GPU system previously, but it was lower end hardware. It was a lot of fun to make it run smoothly with its own challenges. It doesn't take a lot of employee's, we did the above on essentially two technical people. Those same two are working on this business. Finding workloads/jobs is definitely going to be an interesting adventure, that said…

150K GPU with two people... presumably, 8 GPU/host, you had close to 20K servers.

I can't really see how that's achievable with only two people, given the time to install hardware, maintain it, deal with outages and planned maintainence and testing, etc. Note: I worked at Google and interfaced with hwops so I have some real-world experience to compare to.

Building a 150K GPU system without a well-understood customer base seems a bit crazy to me. You will either become a hyperscale, serve a niche, or go out of business, I fear.

Re: Cheyenne Super Computer Auction

#124

Post by latchkey »

dekhn wrote: post #123
latchkey wrote: post #120

Earlier quoted context omitted.

Thanks for the feedback. You make a lot of good points. I've built a 150,000 GPU system previously, but it was lower end hardware. It was a lot of fun to make it run smoothly with its own challenges. It doesn't take a lot of employee's, we did the above on essentially two technical people. Those same two are working on this business. Finding workloads/jobs is definitely going to be an interesting adventure, that said…

150K GPU with two people... presumably, 8 GPU/host, you had close to 20K servers. I can't really see how that's achievable with only two people, given the time to install hardware, maintain it, deal with outages and planned maintainence and testing, etc. Note: I worked at Google and interfaced with hwops so I have some real-world experience to compare to. Building a 150K GPU system without a well-understood customer…

7 separate data centers all around the US.

12 GPU/host. 130,000 of that kind. ~10,833 hosts.

The ASRock BC-250's we deployed were 12 individual blades and those were all PXE booted. We deployed 20,000 of those blades across 2 data centers. This was a massive feat of engineering, especially during covid where I couldn't even access the machine directly. Built a whole dashboard to monitor it all too.

I know, I can't believe we did it either, but we did. Software automation was king. I built a single binary that ran on each individual host and knew how to self configure / optimize everything. Idempotently. Even distributing upgrades to the binary was a neat challenge that I solved perfectly, in very creative ways.

Today, we are starting much smaller. Literally from zero/scratch. Given the cost of MI300x, I doubt we will ever get to 150k GPUs, that's an absurd amount of money, but who knows.

Re: Cheyenne Super Computer Auction

#125

Post by dekhn »

latchkey wrote: post #124
dekhn wrote: post #123

Earlier quoted context omitted.

150K GPU with two people... presumably, 8 GPU/host, you had close to 20K servers. I can't really see how that's achievable with only two people, given the time to install hardware, maintain it, deal with outages and planned maintainence and testing, etc. Note: I worked at Google and interfaced with hwops so I have some real-world experience to compare to. Building a 150K GPU system without a well-understood customer…

7 separate data centers all around the US. 12 GPU/host. 130,000 of that kind. ~10,833 hosts. The ASRock BC-250's we deployed were 12 individual blades and those were all PXE booted. We deployed 20,000 of those blades across 2 data centers. This was a massive feat of engineering, especially during covid where I couldn't even access the machine directly. Built a whole dashboard to monitor it all too. I know, I can't be…

But who did the wiring? Even with blades which consolidate much of the cabling, there's still a tremendous amount of work to build the interconnect. On typical large systems I've seen a small team 3-5 guys working weeks+ to wire a modest DC.

Re: Cheyenne Super Computer Auction

#126

Post by latchkey »

dekhn wrote: post #125
latchkey wrote: post #124

Earlier quoted context omitted.

7 separate data centers all around the US. 12 GPU/host. 130,000 of that kind. ~10,833 hosts. The ASRock BC-250's we deployed were 12 individual blades and those were all PXE booted. We deployed 20,000 of those blades across 2 data centers. This was a massive feat of engineering, especially during covid where I couldn't even access the machine directly. Built a whole dashboard to monitor it all too. I know, I can't be…

But who did the wiring? Even with blades which consolidate much of the cabling, there's still a tremendous amount of work to build the interconnect. On typical large systems I've seen a small team 3-5 guys working weeks+ to wire a modest DC.

We'd hire the initial deployment out to temporary contractors. It just took a few weeks to get a large deployment out. The hard part was the 12 GPUs needed to be inserted at the DC, which took a bunch of effort. Once it was done we generally had 1-2 people on the ground in the data centers to deal with breakfixes. Either contractors or supplied by the DC.

For this venture, again, we are starting small, so we are just flying to the DC and doing it ourselves. There are also staff there that are technical enough to swap stuff out when we need it. The plan will be to just hire one of their staff as our own.

I don't think we will make it for this next deployment due to time constraints, but ideally in our near future, we will go full L11. Assemble and ship out full racks at the manufacturer/VAR, bolt em down, wire them up and ready to go. That is my dream... we will see if we get there. L11 is hard cause a single missing cable can hold up an entire shipment.

Re: Cheyenne Super Computer Auction

#127

Post by dekhn »

jasonwatkinspdx wrote: post #121
dekhn wrote: post #116

Earlier quoted context omitted.

Does 1/64" precision really mean anything in wood, where small fluctuations in air moisture can cause > 1/64" distortion? I guess it's OK if you stay within a climate controlled area.

So he would build parts by first making an oversize rough blank of bonded layers of marine grade plywood in a big press. Then he'd rough cut it various ways on a big band saw. Then he'd work his way through using all the sanders to slowly approach the net shape. He used precision squares to measure bigger stuff and calipers for smaller stuff. I can't tell you the exact stability of marine grade plywood, but I know it…

Plywood is good for dimensional stability, but I'm pretty sure all this work must have been done in and around a toolroom with stable moisture content or the part was used immediately and then consumed/destroyed before it "moves" too much. However, he sounds like he's pretty knowledgeable so I'm going to guess this isn't just garage woodworking where 1/32" doesn't really matter when the wood is going to shrink/expand by 5-10% over the course of a year.

Re: Cheyenne Super Computer Auction

#128

Post by layoric »

NickC25 wrote: post #108

What could someone possibly do with this? It's cool as hell but 8 years old.

I dunno, I still think the 2011-v3 platform that these Xeons can run in is still a great setup for a homelab. A bit power hungry but if you can build a dual core workstation with 36 cores, and 256GB ram for Take the Dell T7910 for example (I use one of these for my homelab), you can pick up a basic one with low end CPU/RAM for sometimes as little as $300. Dumping all these 18 core E5s and DDR4 ECC on the market should make it even cheaper to spec out. Currently they go for about $100-150 each on the CPUs, and ~$150-200 for the RAM. Not bad IMO.

Re: Cheyenne Super Computer Auction

#129

Post by RajT88 »

fnord77 wrote: post #75
mk_stjames wrote: post #27

Earlier quoted context omitted.

Given that the individual nodes are just x86_64 Xeons and run linux... it would be interesting to part it out for sale as individual, but functional, nodes to people. There are a lot of people would like to have a ~2016 era watercooled 1U server from a supercomputer that was once near the top of the Top500 just to show to people. Get little commemorative plaques for each one and sell for $200 each or so. edit: it see…

I don't think there's that big of a market for obsolete server pieces as nostalgia... But you could probably make a decent profit on just the CPUs alone parted out, even with the moving/handling costs.

Going off one listing for a E5-2697v4, $50 with free shipping, 386 already sold.

If you figure after the double-dipping of eBay/Paypal and then shipping fees, that's ~$30 profit per CPU.

8024 x 30 = 241,920 USD. Not too shabby for what's got to be some weeks/months of work. You could probably assume that they can sell or scrap the rest of it for a bit more as well, minus the fees for storage and moving company.

Re: Cheyenne Super Computer Auction

#130

Post by lightedman »

mchannon wrote: post #94

I once bought a far larger supercomputer. It was 1/8 (roughly) of ASCI Blue Mountain. 72 racks. Commissioned in 1998 as #1 or #2 on the TOP500, officially decommissioned in 2004, purchased my 1/8 for $7k in ~2005. Moving 72 racks was NOT easy. After paying substantial storage fees, I rented a 1500sf warehouse after selling off a few of them and they filled it up. Took a while to get 220V/30A service in there to run j…

"The original NFL broadcasts with that magic yellow first down line were synthesized with similar hardware"

That was originally an XFL innovation.

Post reply on HN