Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A day in the life of the fastest supercomputer

nature.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: A day in the life of the fastest supercomputer

#51

Post by wickberg »

7373737373 wrote: post #12

So what is the actual utilization % of this machine?

On a node-level, usually these are aiming for around 90-95% allocated. Note that, compared to most "cloud" applications, that usually involves a number of tricks at the system scheduling level to achieve.

At some point, in order to concurrently allocate a 1000-node job, all 1000 nodes will need to be briefly unoccupied ahead of that, and that can introduce some unavoidable gaps in system usage. Tuning in the "backfill" scheduling part of the workload manager can help reduce that, and a healthy mix of smaller single-node short-duration work alongside bigger multi-day multi-thousand-node jobs helps keep the machine busy.

Re: A day in the life of the fastest supercomputer

#52

Post by p1esk »

wickberg wrote: post #50
jasonwatkinspdx wrote: post #35

Earlier quoted context omitted.

It's not just about the link speeds, it's about the topologies used. Google style infrastructure uses aggregation trees. This works well for fan out fan back in communication patterns, but has limited bisection bandwidth at the core/top of the tree. This can be mitigated with clos networks / fat trees, but in practice no one goes for full bisection bandwidth on these systems as the cost and complexity aren't justifie…

Almost no modern systems are running Torus these days - at least not at the node level. The backbone links are still occasionally designed that way, although Dragonfly+ or similar is much more common and maps better onto modern switch silicon. You're spot on that the bandwidth available in these machines hugely outstrips that in common cloud cluster rack-scale designs. Although full bisection bandwidth hasn't been a…

LambdaLabs GPU cluster provides internode bandwidth of 3.2Tbps: I personally verified it in a cluster of 64 nodes (8xH100 servers) and they claim it holds for up to 5k GPU cluster. What is the internode bandwidth of Frontier? Someone claimed it's 200Gbps, which, if true, would be a huge bottleneck for some ML models.

Re: A day in the life of the fastest supercomputer

#53

Post by wickberg »

dauertewigkeit wrote: post #17

Don't the industry labs have bigger machines by now? I lost track.

"Aurora" at Argonne National Labs is intended to be a bit bigger, but has suffered through a long series of delays. It's expected to surpass Frontier on the TOP500 list this fall once they some issues resolved. El Capitan at LLNL is also expected to be online soon, although I'm not sure if it'll be on the list this fall or next spring.

As others note, these systems are measured by running a specific benchmark - Linpack - and require the machine to be formally submitted. There are systems in China that are on a similar scale, but, for political reasons, have not formally submitted results. There are also always rumors around the scale of classified systems owned by various countries that are also not publicized.

Alongside that, the hyperscale cloud industry has added some wrinkles to how these are tracked and managed. Microsoft occupies the third position with "Eagle", which I believe is one of their newer datacenter deployments briefly repurposed to run Linpack. And they're rolling similar scale systems out on a frequent basis.

Re: A day in the life of the fastest supercomputer

#54

Post by wickberg »

p1esk wrote: post #52
wickberg wrote: post #50

Earlier quoted context omitted.

Almost no modern systems are running Torus these days - at least not at the node level. The backbone links are still occasionally designed that way, although Dragonfly+ or similar is much more common and maps better onto modern switch silicon. You're spot on that the bandwidth available in these machines hugely outstrips that in common cloud cluster rack-scale designs. Although full bisection bandwidth hasn't been a…

LambdaLabs GPU cluster provides internode bandwidth of 3.2Tbps: I personally verified it in a cluster of 64 nodes (8xH100 servers) and they claim it holds for up to 5k GPU cluster. What is the internode bandwidth of Frontier? Someone claimed it's 200Gbps, which, if true, would be a huge bottleneck for some ML models.

Frontier is 4x 200Gbps links per node into the interconnect. The interconnect is designed for 540TB/s of bisection bandwidth. https://icl.utk.edu/files/publications/2022/icl-utk-1570-202...>

Bisection bandwidth is the metric these systems will cite, and impacts how the largest simulations will behave. Inter-node bandwidth isn't a direct comparison, and can be higher at modest node counts as long as you're within a single switch. I haven't seen a network diagram for LambdaLabs, but it looks like they're building off 200Gbps Infiniband once you get outside of NVLink. So they'll have higher bandwidth within each NVLink island, but the performance will drop once you need to cross islands.

Re: A day in the life of the fastest supercomputer

#55

Post by wickberg »

p1esk wrote: post #41
markstock wrote: post #27

Earlier quoted context omitted.

Each node has 4 GPUs, and each of those has a dedicated network interface card capable of 200 Gbps each way. Data can move right from one GPU's memory to another. But it's not just bandwidth that allows the machine to run so well, it's a very low-latency network as well. Many science codes require very frequent synchronizations, and low latency permits them to scale out to tens of thousands of endpoints.

200 Gbps Oh wow, that’s pretty bad.

That's 200Gbps from that card to any other point in the other 9,408 nodes in the system. Including file storage.

Within the node, bandwidth between the GPUs is considerably higher. There's an architecture diagram at https://docs.olcf.ornl.gov/systems/frontier_user_guide.html> that helps show the topology.

Re: A day in the life of the fastest supercomputer

#56

Post by nxobject »

msteffen wrote: post #11
iJohnDoe wrote: post #5

The analogies used in this article were a bit weird. Two things I’ve always wondered since I’m not an expert. 1. Obviously, applications must be written to run effectively to distribute the load across the supercomputer. I wonder how often this prevents useful things from being considered to run on the supercomputer. 2. It always seems like getting access to run anything on the supercomputer is very competitive or ev…

My former employer (Pachyderm) was acquired by HPE, who built Frontier (and sells supercomputers in general), and I’ve learned a lot about that area since the acquisition. One of the main differences between supercomputers and eg a datacenter is that in the former case, application authors do not, as a rule, assume hardware or network issues and engineer around them. A typical supercomputer workload will fail overall…

I wouldn’t be surprised if the nice guarantees given by scientific supercomputers came from the time when mainframes were the only game in town for scientific computing.

Re: A day in the life of the fastest supercomputer

#57

Post by AlotOfReading »

p1esk wrote: post #52
wickberg wrote: post #50

Earlier quoted context omitted.

Almost no modern systems are running Torus these days - at least not at the node level. The backbone links are still occasionally designed that way, although Dragonfly+ or similar is much more common and maps better onto modern switch silicon. You're spot on that the bandwidth available in these machines hugely outstrips that in common cloud cluster rack-scale designs. Although full bisection bandwidth hasn't been a…

LambdaLabs GPU cluster provides internode bandwidth of 3.2Tbps: I personally verified it in a cluster of 64 nodes (8xH100 servers) and they claim it holds for up to 5k GPU cluster. What is the internode bandwidth of Frontier? Someone claimed it's 200Gbps, which, if true, would be a huge bottleneck for some ML models.

It's 200 Gbps per port, per direction. That's the same as the Nvidia interconnect lambdalabs uses.

Re: A day in the life of the fastest supercomputer

#58

Post by p1esk »

wickberg wrote: post #54
p1esk wrote: post #52

Earlier quoted context omitted.

LambdaLabs GPU cluster provides internode bandwidth of 3.2Tbps: I personally verified it in a cluster of 64 nodes (8xH100 servers) and they claim it holds for up to 5k GPU cluster. What is the internode bandwidth of Frontier? Someone claimed it's 200Gbps, which, if true, would be a huge bottleneck for some ML models.

Frontier is 4x 200Gbps links per node into the interconnect. The interconnect is designed for 540TB/s of bisection bandwidth. https://icl.utk.edu/files/publications/2022/icl-utk-1570-202... > Bisection bandwidth is the metric these systems will cite, and impacts how the largest simulations will behave. Inter-node bandwidth isn't a direct comparison, and can be higher at modest node counts as long as you're within a s…

I thought NVLink is only for communication between GPUs within a single node, no? I don't know what the size of their switches are, but I verified that within a 64 node cluster I got the full advertised 3.2Tbps bandwidth. So that's 4x as fast as 4x200Gbps, but 800Gbps is probably not a bottleneck for any real world workload.

Re: A day in the life of the fastest supercomputer

#59

Post by p1esk »

wickberg wrote: post #55
p1esk wrote: post #41

Earlier quoted context omitted.

200 Gbps Oh wow, that’s pretty bad.

That's 200Gbps from that card to any other point in the other 9,408 nodes in the system. Including file storage. Within the node, bandwidth between the GPUs is considerably higher. There's an architecture diagram at https://docs.olcf.ornl.gov/systems/frontier_user_guide.html > that helps show the topology.

I see, OK, I misinterpreted it as per node bandwidth. Yes, this makes more sense, and is probably fast enough for most workloads.

Re: A day in the life of the fastest supercomputer

#60

Post by Enginerrrd »

kaycebasques wrote: post #6

What's the documentation like for supercomputers? I.e. when a researcher gets approved to use a supercomputer, do they get lots of documentation explaining how to set up and run their program? I got the sense from a physicist buddy that a lot of experimental physics stuff is shared informally and never written down. Or maybe each field has a couple popular frameworks for running simulations, and the Frontier people j…

My understanding is that usually there is a subject matter expert that will help you adapt your code to the specific machine to get optimal performance when it's your turn for compute time.
Post reply on HN