21–30 of 125 posts
Earlier quoted context omitted.
Every BLAS you want to use has at least some connection to UT Austin’s TACC.
aren't the lapack people in tennessee?
It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
> I’m also curious if HPC practitioners have to fiddle a lot of black-box knobs to squeeze out performance?
In my experience with CUDA developers, yes the Shmoo Plot (https://en.wikipedia.org/wiki/Shmoo_plot, sometimes called a ‘wedge’ in some industries) is one of the workhorses of every day optimization. I’m not sure I’d call it black-box, though maybe the net effect is the same. It’s really common to have educated guesses and to know what the knobs do and how they work, and still find big surprises when you measure. The first rule of optimization is measure. I always think of Michael Abrash’s first chapter in the “Black Book”: “The Best Optimizer is Between Your Ears” http://twimgs.com/ddj/abrashblackbook/gpbb1.pdf. This is a fabulous snippet of the philosophy of high performance (even though it’s PC game centric and not about modern HPC.)
Related to your point about abstraction, the heaviest knob-tuning should get done at the end of the optimization process, because as soon as you refactor or change anything, you have to do the knob tuning again. A minor change in register spills or cache access patterns can completely reset any fine-tuning of thread configuration or cache or shared memory size, etc.. Despite this, some healthy amount of knob tuning is still done along the way to check & balance & get an intuitive sense of the local perf space of the code. (Just noticed Abrash talks a little about why this is a good idea.)
It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
You won't be able to use a supercomputer at all without scalability, and it's the one topic that is specific to it. But, of course, those computers time is quite expensive so you'll want to optimize for performance too. It's just secondary.
I haven't found many good sources for this kind of information. If you are aware of any, please cite them in a comment below.
It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech wit…
The other cool thing about HPC is it is one of the last areas where multi-user Unix is used! At least, if you're using a university or NSF cluster that is!
Only other place I really see multiple humans using the same machine is SDF or the Tildes
It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech wit…
I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…
I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…
Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.
Earlier quoted context omitted.
You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech wit…
Having switched from LSF to slurm, I have to appreciate that the ecosystem is so bash-centric. Lots of re-use in the conversion. If I’d had to learn some kind of slurm-markup-language or slurmScript or find buttons in some SlurmWizard, it would have been a nightmare.
Fist time I saw it, I had a flashback to the times when I worked for HP, and they were making some huge SAP knock-off, and that system was so labor-intensive to deploy that their QA process involved actual patches. As in pre-release QA cycle involved installing the system, validating it (which could take a few weeks) and if it's not considered DoD, then the developers are given the final list of things they need to fix and those fixes would have to be submitted as patches (sometimes, literal diffs that need to be applied to the deployed system with the patch tool).
This is, I guess, how the "patch version component" came to be in SemVer spec. It's kind of funny how lots of tools are using this component today for completely unrelated purposes... but yeah, LSF feels like the time is ticking there at a different pace :)
It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech wit…