The art of high performance computing

> I mean, memory topology varies greatly by uarch...

You're absolutely right, this is why I said that if you're using libraries, this burden is generally handled by them. Also compilers do this and handle this very well.

If you're writing your own routines, the best way is to read the arch docs, maybe some low-level sites like chips and cheese, do some synthetic benchmarks and write your code in a semi informed way.

After writing the code, a suite of cachegrind, callgrind and perf is on order. See if there are any other bottlenecks, and tune your code accordingly. Add hints for your compiler, if possible.

I was able to reach insane saturation levels with Eigen plus, some hand-tuned code. For the next level, I needed to change my matrix ordering, but it was already fast enough (30 minutes to 45 seconds: 40x speedup), so I left it there.

Sometimes there are no replacement for blood, sweat and tears in this thing.

I have never played with custom interconnects (Slingshot, etc.), yet, so I can't tell much.

> And what's wrong with shell scripts?

When used as configuration? Here are some things that are wrong:

* Configuration forced into a single line makes writing long lines inconvenient (for example, if you want Slurm with Pyxis, and you need to specify the image name -- it will most likely not fit on the screen.

* Oh, and since we mentioning Pyxis -- their image names have pound sign in them, and now you also need to figure out how to escape it, because for some reason if used literally it breaks the comments parser.

* No syntax highlighting (because it's all comments).

* No way to create more complex configuration, i.e. no way to have any types other than strings, no way to have variables, no way to have collections of things.

* No way to reuse configuration (you have to copy it from one job file to another). I honestly don't even know what happens if you try to source a job configuration file from another job configuration.

All in all, it's really hard to imagine a worse configuration format. This sounds like a solution from some sort of a code-golfing competition where the goal was to make it as bad as possible, while still retaining some shreds of functionality.

> proportional to the fourth power of the temperature difference between the body and surroundings

Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is one fin of the heatsink radiating right at the next one).

So cooling in space is hard. You’re not getting GPU-like power densities without a physically immense radiating surface extending way past those GPUs.

[0] Caveat: emissivity can depend on wavelength, and the law holds independently at each wavelength. So this can introduce interesting effects, which is how all the fancy prototype roof-cooling materials work, and it’s also related to how “spectrally selective” windows and window films work.

Thanks. I have read that paper!

The real world is way more complicated...

You can think about each host as its own markov chain: they may be serving, have hidden or diagnosed problems at various confidence levels, be on route to various remedial processes (reboot/reinstall/repairs I cited above, to simplify), require software/firmware changes, scheduled or opportunistic diagnostics (e.g. periodic deep-screening for otherwise silent problems).

Repair workflows are even more complicated and depend on the specific of the fault, and connection of components (e.g. a system diagnosed with a missing GPU may be because a cable or an interposer card is not seated correctly). Also, repair time is modulated by work shifts and some details of the logistics in datacenters.

Parts can become bottlenecks too. I remember one time in the early TPUv3 days: we had delays in recovering from a large incident because of fault-positive diagnoses of a $4 fan, that was widely used in all systems.

Add that nowadays systems are not one host and some attached cards, but have multiple nodes: e.g. the simplest combination is a main compute node and the smart nic / IPU, but some systems can be a lot more complicated.

So these alone are a hierarchical markov chains, with inhomogeneous arrival times and service times that are themselves time-dependent functions. There are also a lot of long-term memory effects, ensemble average is not necessarily the the time average. Chaotic behavior, in the mathematical sense, is common.

Systems are built with field replaceable units (FRU): e.g. in some generations, you can't swap one GPU in a server that has 8, you have the swap a whole block of them. You can choose when to repair, to maximize TCO/$ based on usage patterns (how many users want all GPUs vs. a smaller number).

Some systems (e.g. TPU pods) have links between accelerator trays, both within a physical rack and across them. So the usefulness of the neighbor system is reduced while a host/tray is being services, and you can have the equivalent of deadlock/livelock in repair dependencies.

Cluster scheduling and management is designed to maximize service levels, minimizing disruptions. This also implies that some disruptions (e.g. when you do some repairs) is driven by what workloads you run. Workloads are power-law distributed in size, so you have small-world network dynamics and the potential for supercritical behavior: both in disruptions (picture jobs preempting other jobs as a graph) but also in risks (add to the previous picture, one job that trigger an hardware problem).

Multiply these by several thousands to get the size of a datacenter cluster. Add supporting compute+storage, networking, power and thermal constraints. Multiply these by the number of clusters (Some hyperscalers have global scheduling systems, that make it possible to see the whole ML fleet as one). Add rare events, because at this scale you start to think about utility and electrical grid failures.

Figuring out what to do in control systems for the current ML fleet is one problem. Simulating what kind of service the future systems a few generations of hardware, software, datacenter design down the line should provide, so you can define what you can offer, influence the designs and make the right investments... it's more complicated. Both of these two problems are my current job.

I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the density and property of air cool in general but ignoring the environment of the machine such as the ambient weather.)

The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine.

The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.