Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A day in the life of the fastest supercomputer

nature.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: A day in the life of the fastest supercomputer

#21

Post by rcxdude »

cubefox wrote: post #20

> With its nearly 38,000 GPUs, Frontier occupies a unique public-sector role in the field of AI research, which is otherwise dominated by industry. Is it really realistic to assume that this is the "fastest supercomputer"? What are estimated sizes for supercomputers used by OpenAI, Microsoft, Google etc? Strangely enough, the Nature piece only mentions possible secret military supercomputers, but not ones used by AI…

There is a difference between a supercomputer and just a large cluster of compute nodes: mainly this is in the bandwidth between the nodes. I suspect industry uses a larger number of smaller groups of highly-connected GPUs for AI work.

Re: A day in the life of the fastest supercomputer

#22

Post by nradclif »

7373737373 wrote: post #12

So what is the actual utilization % of this machine?

I don’t know the exact utilization, but most large supercomputers that I’m familiar with have very high utilization, like around 90%. The Slurm/PBS queue times can sometimes be measured in days.

Re: A day in the life of the fastest supercomputer

#23

Post by p1esk »

rcxdude wrote: post #21
cubefox wrote: post #20

> With its nearly 38,000 GPUs, Frontier occupies a unique public-sector role in the field of AI research, which is otherwise dominated by industry. Is it really realistic to assume that this is the "fastest supercomputer"? What are estimated sizes for supercomputers used by OpenAI, Microsoft, Google etc? Strangely enough, the Nature piece only mentions possible secret military supercomputers, but not ones used by AI…

There is a difference between a supercomputer and just a large cluster of compute nodes: mainly this is in the bandwidth between the nodes. I suspect industry uses a larger number of smaller groups of highly-connected GPUs for AI work.

Do you mean this supercomputer has slower internode links? What are its links? For example, xAI just brought up 100k GPU cluster, most likely with 800Gbps internode links, or maybe even double that.

I think the main difference is in the target numerical precision: supercomputers such as this one focus on maximizing FP64 throughput, while GPU clusters used by OpenAI or xAI want to compute in 16 or even 8 bit precision (BF16 or FP8).

Re: A day in the life of the fastest supercomputer

#24

Post by markstock »

kaycebasques wrote: post #6

What's the documentation like for supercomputers? I.e. when a researcher gets approved to use a supercomputer, do they get lots of documentation explaining how to set up and run their program? I got the sense from a physicist buddy that a lot of experimental physics stuff is shared informally and never written down. Or maybe each field has a couple popular frameworks for running simulations, and the Frontier people j…

https://docs.olcf.ornl.gov/systems/frontier_user_guide.html

This will have much of what you need.

Re: A day in the life of the fastest supercomputer

#25

Post by markstock »

tryauuum wrote: post #8
iJohnDoe wrote: post #5

The analogies used in this article were a bit weird. Two things I’ve always wondered since I’m not an expert. 1. Obviously, applications must be written to run effectively to distribute the load across the supercomputer. I wonder how often this prevents useful things from being considered to run on the supercomputer. 2. It always seems like getting access to run anything on the supercomputer is very competitive or ev…

I feel like the name "supercomputer" is overhyped. It's just many normal x86 machines running Linux and connected with fast network. Here in Finland I think you can use LUMI supercomputer for free. With a condition that the results should be publically available

FYI: LUMI uses a nearly identical architecture as Frontier (AMD CPUs and GPUs), and was also made by HPE.

Re: A day in the life of the fastest supercomputer

#27

Post by markstock »

p1esk wrote: post #23
rcxdude wrote: post #21

Earlier quoted context omitted.

There is a difference between a supercomputer and just a large cluster of compute nodes: mainly this is in the bandwidth between the nodes. I suspect industry uses a larger number of smaller groups of highly-connected GPUs for AI work.

Do you mean this supercomputer has slower internode links? What are its links? For example, xAI just brought up 100k GPU cluster, most likely with 800Gbps internode links, or maybe even double that. I think the main difference is in the target numerical precision: supercomputers such as this one focus on maximizing FP64 throughput, while GPU clusters used by OpenAI or xAI want to compute in 16 or even 8 bit precision…

Each node has 4 GPUs, and each of those has a dedicated network interface card capable of 200 Gbps each way. Data can move right from one GPU's memory to another. But it's not just bandwidth that allows the machine to run so well, it's a very low-latency network as well. Many science codes require very frequent synchronizations, and low latency permits them to scale out to tens of thousands of endpoints.

Re: A day in the life of the fastest supercomputer

#28

Post by ai_slurp_bot »

kkielhofner wrote: post #9

I have a project on Frontier - happy to answer any questions! Funny story about Bronson Messer (quoted in the article): On my first trip to Oak Ridge we went on a tour of “The Machine”. Afterwards we were hanging out on the observation deck and got introduced to something like 10 people. Everyone at Oak Ridge is just Tom, Bob, etc. No titles or any of that stuff - I’m not sure I’ve ever heard anyone refer to themselv…

Hey, my sister Katie is the reason he wasn't a 4 day champ! Beat him by $1. She also lost her next game

Re: A day in the life of the fastest supercomputer

#29

Post by dmd »

johnklos wrote: post #16
kkielhofner wrote: post #9

I have a project on Frontier - happy to answer any questions! Funny story about Bronson Messer (quoted in the article): On my first trip to Oak Ridge we went on a tour of “The Machine”. Afterwards we were hanging out on the observation deck and got introduced to something like 10 people. Everyone at Oak Ridge is just Tom, Bob, etc. No titles or any of that stuff - I’m not sure I’ve ever heard anyone refer to themselv…

> anyone refer to themselves or anyone else as "Doctor". Reminds me of the t-shirt I had that said, "Ok, Ok, so you've got a PhD. Just don't touch anything."

I think when I walked back into my defense and they said "congratulations, Doctor Drucker" was the last time anyone ever called me Doctor except for possibly a hotel clerk when I selected 'Dr' as my honorific.

It's just not in the culture, assuming you mostly work among other PhDs.

Re: A day in the life of the fastest supercomputer

#30

Post by CaliforniaKarl »

langcss wrote: post #26

Or worlds smallest cloud provider?

That's a bit of an apples-and-oranges comparison. Cloud services normally have different design goals.

HPC workloads are often focused on highly-parallel jobs, with high-speed and (especially) low-latency communications between nodes. Fun fact: In the NVIDIA DGX SuperPOD Reference Architecture, each DGX H100 system (which has eight H100 GPUs per system) has four Infiniband NDR OSFP ports dedicated to GPU traffic. IIRC, each OSFP port operates at 200 Gbps (two lanes of 100 Gbps), allowing each GPU to effectively have its own IB port for GPU-to-GPU traffic.

(NVIDIA's not the only group doing that, BTW: Stanford's Sherlock 4.0 HPC environment[2], in their GPU-heavy servers, also uses multiple NDR ports per system.)

Solutions like that are not something you'll typically find in your typical cloud provider.

Early cloud-based HPC-focused solutions centered on workload locality, not just within a particular zone but with a particular part of a zone, with things like AWS Placement Groups[3]. More-modern Ethernet-based providers will give you guides like [4], telling you how to supplement placement groups with directly-accessible high-bandwidth network adapters, and in particular support for RDMA [4] or RoCE (RDMA over Converged Ethernet), which aims to provide IB-like functionality over Ethernet.

IMO, the closest analog you'll find in the cloud, to environments like Frontier, is going to be IB-based cloud environments from Azure HPC ('general' cloud) [5] and specialty-cloud folks like Lambda Labs [6].

[1]: https://docs.nvidia.com/dgx-superpod/reference-architecture-...

[2]: https://news.sherlock.stanford.edu/publications/sherlock-4-0...

[3]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen...

[4]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html

[5]: https://azure.microsoft.com/en-us/solutions/high-performance...

[6]: https://lambdalabs.com/nvidia/dgx-systems

Post reply on HN