El Capitan: New supercomputer is the fastest • Hacker News Board

Re: El Capitan: New supercomputer is the fastest

#41

Post by philipkglass » Tue, Nov 19, 2024, 9:51 PM UTC

declan_roberts wrote: post #32

Do super computers need proximity to other compute nodes in order to perform this kind of computations? I wonder what would happen if Apple offered people something like iCloud+ in exchange for using their idle M4 compute at night time for a distributed super computer.

Yes, supercomputers need low-latency communication between nodes. If a problem is "embarrassingly parallel" (like folding@home, mentioned by sibling comment) then you can use loosely coordinated nodes. Those sorts of problems usually don't run on supercomputers in the first place, since there are cheaper ways to solve them.

Re: El Capitan: New supercomputer is the fastest

#42

Post by balia » Tue, Nov 19, 2024, 9:53 PM UTC

Some may not want to hear this, but these “fastest supercomputer” list is now meaningless because all the Chinese labs have started obfuscating their progress.

A while ago there were a few labs in China in top 10 and they all attracted sanctions / bad attention. Now no Chinese lab report any data now

Re: El Capitan: New supercomputer is the fastest

#43

Post by Koshkin » Tue, Nov 19, 2024, 9:53 PM UTC

alephnerd wrote: post #26

pama wrote: post #23

Noting here that 2700 quadrillion operations per second is less than the estimated sustained throughput of productive bfloat16 compute during the training of the large llama3 models, which IIRC was about 45% of 16,000 quadrillion operations per second, ie 16k H100 in parallel at about 0.45 MFU. The compute power of national labs has fallen far behind industry in recent years.

Training an LLM (basically Transformers) is different workflow from Nuclear Simulations (basically Monte Carlo simulations) There are a lot of intricates, but at a high level they require different compute approaches.

This is about the raw compute, no matter the workflow.

Re: El Capitan: New supercomputer is the fastest

#44

Post by Koshkin » Tue, Nov 19, 2024, 9:54 PM UTC

bryanlarsen wrote: post #30

pama wrote: post #23

Noting here that 2700 quadrillion operations per second is less than the estimated sustained throughput of productive bfloat16 compute during the training of the large llama3 models, which IIRC was about 45% of 16,000 quadrillion operations per second, ie 16k H100 in parallel at about 0.45 MFU. The compute power of national labs has fallen far behind industry in recent years.

A 64 bit float operation is >4X as expensive as a 16 bit float operation.

In terms of heat dissipation, maybe, yes. But not necessarily in time.

Re: El Capitan: New supercomputer is the fastest

#45

Post by pama » Tue, Nov 19, 2024, 9:54 PM UTC

bryanlarsen wrote: post #30

pama wrote: post #23

Noting here that 2700 quadrillion operations per second is less than the estimated sustained throughput of productive bfloat16 compute during the training of the large llama3 models, which IIRC was about 45% of 16,000 quadrillion operations per second, ie 16k H100 in parallel at about 0.45 MFU. The compute power of national labs has fallen far behind industry in recent years.

A 64 bit float operation is >4X as expensive as a 16 bit float operation.

Agreed. However also note that if it was only matrix multiplies and no full transformer training, the performance of that Meta cluster would be closer to 16k PFlops/s, still much faster than the El Capitain performance measured on linpack and multiplied by 4. Other companies presumably cabled 100k H100s together, but they dont yet publish training data for their LLMs. It is good to have competition, I just didnt expect the tables to flip so dramatically over the last two decades from a time when governments still ruled the top spots in computer centers with ease to nowadays where the assumption is that there are at least ten companies with larger clusters than the most powerful governments.

Re: El Capitan: New supercomputer is the fastest

#46

Post by pertymcpert » Tue, Nov 19, 2024, 9:55 PM UTC

pama wrote: post #31

alephnerd wrote: post #26

Earlier quoted context omitted.

Training an LLM (basically Transformers) is different workflow from Nuclear Simulations (basically Monte Carlo simulations) There are a lot of intricates, but at a high level they require different compute approaches.

Absolutely. Though the performance of El Capitain is only measured by a linpack benchmark not the actual application.

I thought modern supercomputers use benchmarks like HPCG instead of LINPACK?

Re: El Capitan: New supercomputer is the fastest

#47

Post by pertymcpert » Tue, Nov 19, 2024, 9:57 PM UTC

handfuloflight wrote: post #29

alephnerd wrote: post #26

Earlier quoted context omitted.

Training an LLM (basically Transformers) is different workflow from Nuclear Simulations (basically Monte Carlo simulations) There are a lot of intricates, but at a high level they require different compute approaches.

Can you expand on why the operations per second is not an apt comparison?

When you're doing scientific simulations, you're generally a lot more sensitive to FP precision than ML training which is very, very tolerant of reduced precision. So while FP8 might be fine for transformer networks, it would likely be unacceptably inaccurate/unusable for simulations.

Re: El Capitan: New supercomputer is the fastest

#48

Post by pknomad » Tue, Nov 19, 2024, 9:58 PM UTC

balia wrote: post #42

Some may not want to hear this, but these “fastest supercomputer” list is now meaningless because all the Chinese labs have started obfuscating their progress. A while ago there were a few labs in China in top 10 and they all attracted sanctions / bad attention. Now no Chinese lab report any data now

I wouldn't say meaningless... just incomplete.

Re: El Capitan: New supercomputer is the fastest

#49

Post by p_l » Tue, Nov 19, 2024, 9:58 PM UTC

pkaye wrote: post #27

p_l wrote: post #15

Earlier quoted context omitted.

It literally requires simulating each subatomic particle, individually. The increases of compute power have been used for twin goals of reducing simulation time (letting you run more simulations) and to increase the size and resolution. The alternative is to literally build and detonate a bomb to get empirical data on given design, which might have problems with replicability (important when applying the results to r…

Are they always designing new nuclear bombs? Why the ongoing work to simulate?

Because even normal explosives degenerate over time, and fissile material in nuclear devices is even worse about it - remember that unstable elements are ongoing constant fission events, critical mass is just one where they trigger each others' fission fast enough for runaway process.

So in order to verify that the weapons are still useful and won't fail in random ways, you have to test them.

Which either involves actually exploding them (banned by various treaties that have enough weight that even USA doesn't break them), or numerical simulations.

Re: El Capitan: New supercomputer is the fastest

#50

Post by JumpCrisscross » Tue, Nov 19, 2024, 9:59 PM UTC

shagie wrote: post #17

einpoklum wrote: post #4

So, they built this supercomputer to test new and more deadly nuclear weapons. That makes me so "happy". I am absolutely not worried about two nuclear powers being close to the brink of direct war, even as we speak; nor about the abandonment of the course of nuclear disarmament treaty; nor about the repeated talk of a coming war against certain Asian powers. Everything is great and I'll just fawn over the colorful li…

I would reference an older article on super computers and the nuclear weapon arsenal. https://www.techtarget.com/searchdatacenter/news/252468294/C... > "The Russians are fielding brand new nuclear weapons and bombs," said Lisa Gordon-Hagerty, undersecretary for nuclear security at the DOE. She said "a very large portion of their military is focused on their nuclear weapons complex." > It's the same for China, which i…

> to see if they still work. It is simulating them to have the confidence that they still work

The Armageddon scenario is some nuclear states conduct stockpile stewardship, some don’t, and those who do discover that warheads come with a use-by date.