Live data from Hacker News

AMD-powered Frontier supercomputer breaks the exascale barrier

tomshardware.com

61–70 of 172 posts

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#61
post #5

Since they are using AMD's accelerators as well [1], I do wonder whether any usage of these will trickle down and give us improvements in ROCm. Surely the people at these labs will want to run ordinary DL frameworks at some point - or do they have the money and time to always build entirely custom stacks? [1] AMD Instinct MI250x in this case.

I’m not using Frontier, but I am using Setonix which is a large AMD cluster being rolled out in Australia. All of AMD’s teaching materials are about ROCm so this is very much how they’re expecting it to be used. The real pain for us is that there’s no decent consumer grade chips with ROCm compatibility for us to do development on. AMD have made it very clear they only care about the data centre hardware when it comes…

Vega64 or Vega56 seems to work pretty well with ROCm in my experience.

Hopefully AMD gets the Rx 6800xt working with ROCm consistently, but even then, the 6800xt is RDNA2, while the supercomputer Mx250x is closer to the Vega64 in more ways.

So all in all, you probably want a Vega64, Radeon VII, or maybe an older MI50 for development purposes.

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#62
post #45
post #25

Earlier quoted context omitted.

The coherent memory interconnects between nodes is typically what makes supercomputers different than just a bunch of consumer hardware. It allows different types of programming or at least makes them easier.

It's a very fast, very low latency network fabric. But it's not coherent in the sense of cache coherent multiprocessors, and it doesn't offer shared memory style programming where you'd just load/store to addresses that happen to be mapped to another compute node somewhere in the system.

I thought DMI allowed for exactly those kinds of load/store operations

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#63
post #36

Earlier quoted context omitted.

I guarantee you an FX-8370 isn't even close to being the fastest CPU even at 10 GHz. I bet most desktop CPUs you can buy nowadays will be faster out of the box.

Tell me what your measure of fast is?

Does it matter? A modern CPU at 5.5ghz will outperform an 8 year old CPU overlocked to 10ghz on just about any reasonable workload even if it’s single threaded.

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#64
I wonder if having one supercomputer with x number of chips or having eight supercomputers each with x/8 number of chips would be the more practical working setup. Weather forecasting for example is basically a complex probabilistic algorithm, and there's a notion that running eight models in parallel and then comparing and contrasting the results will give better estimates of actual outcomes than running one model on a much more powerful machine.

Is it feasible to run eight models on one supercomputer, or is that inefficient?

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#66

Congratulations to AMD, HPE and ORNL! This is an amazing achievement. Can't wait to see the spectacular science results coming from this installation. Intel was supposed to build the first Exascale system for ANL [1] [2]. to be installed by 2018. They completely and utterly messed up the execution, partly drive by 10nm failure, went back to the drawing board multiple times, and now Raja switched the whole thing to GP…

I worked at Intel in a very closely related area.

I quit after getting vaccinated for COVID, only stayed because of the pandemic.

The biggest problem was that Intel simply couldn't execute. They couldn't design and manufacture hardware in a timely manner without too many bugs. I think this was due to poor management practices. My direct manager was amazing, but my skiplevel was always dealing with fires. It felt like instead of the effort being orchestrated that someone approached a crowd of engineers and used a bullhorn to tell them the big goal and that was it. The left hand had no idea what the right hand was doing.

I often called Intel an 'ant hill', because the engineers would swarm a project just like ants do a meal. Some would get there and pull the project forward, some would get on top and uselessly pull upward, and more than I'd like would get behind the project and pull it backwards. Just a mindless swarm of effort, which generally inefficiently kinda did the right thing sometimes.

The inability to execute started to effect my work. When I got a ticket to complete something, I just wouldn't. There was a very good chance that I'd have an extra few weeks (due to slippage) or the task would never need to get done, because the hardware would never appear. Planning was impossible.

Conversely, sometimes hardware CAME OUT OF NOWHERE, not simple stuff, but stuff like laptops made by partners. Just randomly my manager would ask me to support a product we were told directly wouldn't exist, but now did. I needed to help our partner with support right now. Our partners were starting to hate us and it was palpable in meetings.

I'm so glad I quit, I was being worked to the bone on a project which will probably fail and be a massive liability. Even if the economy crashes, and I can't get a job for years, and end up broke, it'll still have been worth it. I also only made 110K/yr base.

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#67
post #65

This reads more or less like a corporate press release - (edit: actually, it reads exactly like a corporate press release) - is there a more substantive article on the topic?

It's not an article, but there's always the front page for the supercomputer (includes some limited specs):

https://www.olcf.ornl.gov/frontier/

There's also detailed architecture specs on Crusher, an identical (but smaller) system:

https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide...

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#68

I wonder if having one supercomputer with x number of chips or having eight supercomputers each with x/8 number of chips would be the more practical working setup. Weather forecasting for example is basically a complex probabilistic algorithm, and there's a notion that running eight models in parallel and then comparing and contrasting the results will give better estimates of actual outcomes than running one model o…

You can run many programs on one supercomputer simultaneously, yes. Check out XSEDE. Cost-wise one big is going to be cheaper than 8 small due to infrastructure issues - cooling, maintenance, space, etc.

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#69
post #55

Earlier quoted context omitted.

Also, while NVidia has been adding extra AI acceleration to their chips AMD has been throwing in extra double precision resources that HPC generally requires. If you're training an AI rather than simulating the climate/a thermonuclear explosion/etc then you're probably better off using NVidia cards but AMD made the right technical investments to get these supercomputer contracts.

It's kind of surprising that nvidia hasn't purchased AMD. It really feels like there's a single company between the two that would be truly effective- AMD for the classic CPU oomph, nvidia for the GPU oomph, combining their strengths in interconnects. It would be a player from the high-end PC to the supercomputer market, without even pretending to go for the low-power market (ARM).

I think based on recent history you can argue that NVIDIA is very aware of the potential anticompetitive actions that could result if they kill or even substantially pass AMD.

There really used to be a lot of intra-generational tweaking and refinement, like if you look back at Maxwell there were really at least 3 and I suspect 4 total steppings of the maxwell architecture (GM107, GM204/GM200, and GM206 - and I suspect GM200 was a separate "stepping" too due to how much higher it clocks than GM204 - which is the opposite of what you'd expect from a big chip). Kepler had at least 4 major versions (GK1xx, GK110B, GK2xx, GK210), Fermi had at least 2 (although that's where I'm no longer super familiar with the exact details).

Anyway point is there used to be a lot more intra-generational refinement, and I think that has largely stopped, it's just thrown over the wall and done. And I think the reason for that is that if NVIDIA really cranked full-steam ahead they'd be getting far enough ahead of AMD to potentially start raising antitrust concerns. We are now in the era of "metered performance release", just enough to stay ahead of AMD but not enough to actually raise problems and get attention from antitrust regulators.

Same thing for the choice of Samsung 8nm for Ampere and TSMC 12nm for Turing, while AMD was on TSMC 7nm for both of those. Sure, volume was a large part of that decision, but they're already matching AMD with a 1-node deficit (Samsung 8nm is a 10+, and the gap between 10 and TSMC 7 is huge to begin with) and they were matching with a 1.5 node deficit during the Turing generation (12FFN is a TSMC 16+ node - that is almost 2 full nodes to TSMC 7nm). They cannot just make arbitrarily fast processors that dump on AMD, or regulators will get mad, so in that case they might as well optimize for cost and volume instead. If they had done a TSMC 7nm against RDNA1 they probably would be starting to get in that danger zone - I'm sure they were watching it carefully during the Maxwell era too.

(the people who imagined some giant falling-out between TSMC are pretty funny in hindsight. (A) NVIDIA still had parts at TSMC anyway, and (B) TSMC obviously couldn't have provided the same volume as Samsung did, certainly not at the same price, and volume ended up being a godsend during the pandemic shortages and mining. Yeah, shortages sucked, but they could still have been worse if NVIDIA was on TSMC and shipping half or 2/3rds of their current volume.)

Of course now we may see that dynamic flip with AMD moving to MCM products earlier, or maybe that won't be for another year or so yet rumors are suggesting monolithic midrange chips will be AMD's first product. Or perhaps "monolithic", being technically MCM but with cache dies/IO dies rather than multiple compute dies. But with RDNA3 AMD is potentially poised to push NVIDIA a little bit, rather than just the controlled opposition we've seen for the past few generations, hence NVIDIA reportedly moving to TSMC N5P and going quite large with a monolithic chip to compete.

Re: AMD-powered Frontier supercomputer breaks the exascale barrier

#70

Earlier quoted context omitted.

> Given all the talk about OpenMP compatibility and Fortran... my guess is that they're largely running legacy code in Fortran. The must used linear algebra library is written in Fortran. There's nothing "legacy" about it, it's just that nobody was able to replicate its speed in C.

> The must used linear algebra library is written in Fortran. My understanding is that most supercomputers have the vendor provide their implementation of BLAS (e.g., if it's Intel-based, you're getting MKL) that's specifically tuned for that hardware. And these implementations stand a decent chance of being written in assembly , not Fortran.

Usually C or Fortran superstructure, and assembly kernels.

The clearest form of this is in BLIS, which is a C framework you can drop your assembly kernel into, and then it makes a BLAS (along with some other stuff) for you. But the idea is also present in OpenBlas.

Lots of this is due to the legacy of gotoBlas (which was forked into OpenBlas, and partially inspired BLIS), written by the somewhat famous (in HPC circles at least) Kazushige Goto. He works at Intel now, so probably they are doing something similar.

Post reply on HN