81–90 of 90 posts
> Why aren’t these AI companies submitting to the TOP500 to show off their computing prowess? my knowledge is 10+ years out of date, but once upon a time if they'd chosen to, Google could have had _several_ entries in the top 10 of the TOP500 list It's just poker, they didn't want to tip their hand
Earlier quoted context omitted.
TOP500 can be done with inexpensive silicon. It is more about a willingness to aggregate enough hardware in one place. As a benchmark, it tells you almost nothing about computing power or scalability for other applications because it doesn't exercise the bottlenecks most high-scale applications have.
> TOP500 can be done with inexpensive silicon. Didn't the DoD at one point build a 1k+ PS3 cluster based on their multi-core chip and had a mini supercomputer CotS? I remember Sony not liking that people were buying them for other things rather than gaming (iirc they were losing money on hardware at the time) so they bricked linux support soon after.
> Why aren’t these AI companies submitting to the TOP500 to show off their computing prowess? my knowledge is 10+ years out of date, but once upon a time if they'd chosen to, Google could have had _several_ entries in the top 10 of the TOP500 list It's just poker, they didn't want to tip their hand
My sense is you only submit if you are in the business of selling supercomputing cluster (IBM, Cray). If you are a consumer or build to consume internally, you would care less.
Earlier quoted context omitted.
Why would the scientific computing people want to tip their hand? It’s an open secret that the main point of these mammoth FP64 compute machines is to simulate nuclear weapons detonations to comply with the CTBT you’d think that crowd would really not be fans of broadcasting their capabilities.
In adversarial scenarios, there are varying strategies in communicating one's capabilities, just as one might do in a poker game. Sometimes you want to show off what you can do to dissuade others from fucking with you. Sometimes you want to undersell your capabilities to hide your true ability. Sometimes you want others to think you are underselling your capabilities when you are actually at a disadvantage.
Earlier quoted context omitted.
My sense is you only submit if you are in the business of selling supercomputing cluster (IBM, Cray). If you are a consumer or build to consume internally, you would care less.
It's also good for recruiting. They're building a new supercomputer in Edinburgh and I'd imagine it's a pull for certain researchers.
Earlier quoted context omitted.
I’ve worked on several systems that had enough flop/s to make it in the top 5-10, but for which we never submitted benchmarks. Sometimes their backend network layout technically would make them several smaller clusters for an HPL run, sometimes it’s because the cluster is too heterogeneous to get a good benchmark result, and sometimes it’s because the employer wants to keep a low profile. Most of the time, it just th…
What programs were yours running to print money?
Earlier quoted context omitted.
> in the best cases one may reach something like 50% to 60% of the maximum If all you need to do is matmuls then you can definitely go past this
You can go past this only on a matmul benchmark, which is seldom useful per se. Linpack consists mostly of matmuls, but nonetheless there are additional operations that prevent GPUs to reach the high utilization of over 80% that is normal for CPUs, so that a throughput over 50% is considered good at the scale of supercomputers. At the scale of a supercomputer, the utilization factor is considerably less than for an i…
Many systems have the node count to be able to run such benchmarks, but are not optimized or even capable of running them. Having these systems run these large calculations in a sustained way, and performing, is what separates a bunch of nodes together in a data center from an actual cluster that is able to run a benchmark like HPL or HPCG.
To sustain 70 - 80% of peak performance across hundreds of thousands of cores, you need a real low-diameter, high-bandwidth, low-latency fabric and a balanced memory subsystem, running on a system w almost no failures or network issues. A loosely-coupled cluster with an oversubscribed fat-tree will 'run Linpack' and then post an Rmax that's a small fraction of its naive peak.
Also, have a look at the Green500, and the systems there. This is not about bragging rights vendors, this is about placing commodity hardware, tuning it, and bringing it up to health in a way that squeezes all of that last performance possible out of the clusters on those lists. That's the opposite of vendor flexing - it's a craft that you cannot see in a simple node count, as some have been comparing here.
If you ever worked on this field, and with the vendors at this scale, you would know. Its not easy, its actually very hard.
... and imagine you need to deploy this, systems at this scale, w technologies that are sometimes just emerging and sometimes even need proper field testing * every 6 months * to be able to reach the scale and stability to land on these lists.
Earlier quoted context omitted.
Yeah, that's true. GPU flops are really impressive in fp16, and more recently fp8 and fp4. When the 40xx GPUs came out, Tim Dettmers had a really cool blog looking into the numbers, and for fp8 one 4090 GPU had enough flops to match the best supercomputer somewhere in the 2000s, with an 8x 4090 build being top for a few years more. It's insane. But it gets nowhere close on fp64, where most of the physics simulations…
Other than weather simulations and nuclear explosions, what other supercomputer tasks are out there?