Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The art of high performance computing

theartofhpc.com

Post reply on HN
Print view View original

11–20 of 125 posts

Re: The art of high performance computing

#11

Post by MichaelZuo »

dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

Memory architecture and bandwidth are still very important, most of IBM's latest performance gains for both mainframes and POWER are reliant on some novel innovations there.

Re: The art of high performance computing

#13

Post by eslaught »

dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

No, the abstractions are not sufficient. We do care about these details, a lot.

Of course, not every application is optimized to the hilt. But if you want to so optimize an application, exactly things you're talking about are what come into play.

So yes, I would expect every competent HPC practitioner to have a solid (if not necessarily intimate) grasp of hardware architecture.

Re: The art of high performance computing

#14

Post by bee_rider »

atrettel wrote: post #10
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

Yes and no. MPI and OpenMP are the primary abstractions from the hardware in HPC, with MPI being an abstracted form of distributed-memory parallel computing and OpenMP being an abstracted form of shared-memory parallel computing. Many researchers write their codes purely using those, often both in the same code. When using those, you really do not need to worry about the architectural details most of the time. Still,…

Of course, not every problem can be solved by BLAS, but if you are doing linear algebra, the cache stuff should be mostly handled by BLAS.

I’m not sure how much multiplication vs addition matters on a modern chip. You can have a bazillion instructions in flight after all, as long as they don’t have any dependencies, so I’d go with whichever option shortens the data dependencies on the critical path. The computer will figure out where to park longer instruction if it needs to.

Re: The art of high performance computing

#16

Post by mgaunard »

dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

Regardless of what you do, domain knowledge tends to be more valuable than purely technical skills.

Knowing more numerical analysis will get probably get you further in HPC than knowledge of specific hardware architectures.

Ideally you want both, of course.

Re: The art of high performance computing

#17

Post by jandrewrogers »

dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

For most HPC, you will not be able to maximize parallelism and throughput without intimate knowledge of the hardware architecture and its behavior. As a general principle, you want the topology of the software to match the topology of the hardware as closely as possible for optimal scaling behavior. Efficient HPC software is strongly influenced by the nature of the hardware.

When I wrote code for new HPC hardware, people were always surprised when I asked for the system hardware and architecture docs instead of the programming docs. But if you understood the hardware design, the correct way of designing software for it became obvious from first principles. The programming docs typically contained quite a few half-truths intended to make things seem misleadingly easier for developers than a proper understanding would suggest. In fact, some HPC platforms failed in large part because they consistently misrepresented what was required from developers to achieve maximum performance in order to appear "easy to use", and then failing to deliver the performance the silicon was capable of if you actually wrote software the way the marketing implied would be effective.

You can write HPC code on top of abstractions, and many people do, but the performance and scaling losses are often unavoidably integer factor. As with most software, this was considered an acceptable loss in many cases if it allowed less capable software devs to design the code. HPC is like any other type of software in that most developers that notionally specialize in it struggle to produce consistently good results. Much of the expensive hardware used in HPC is there to mitigate the performance losses of worse software designs.

In HPC there are no shortcuts to actually understanding how the hardware works if you want maximum performance. Which is no different than regular software, in HPC the hardware systems are just bigger and more complex.

Re: The art of high performance computing

#18

Post by atrettel »

bee_rider wrote: post #14
atrettel wrote: post #10

Earlier quoted context omitted.

Yes and no. MPI and OpenMP are the primary abstractions from the hardware in HPC, with MPI being an abstracted form of distributed-memory parallel computing and OpenMP being an abstracted form of shared-memory parallel computing. Many researchers write their codes purely using those, often both in the same code. When using those, you really do not need to worry about the architectural details most of the time. Still,…

Of course, not every problem can be solved by BLAS, but if you are doing linear algebra, the cache stuff should be mostly handled by BLAS. I’m not sure how much multiplication vs addition matters on a modern chip. You can have a bazillion instructions in flight after all, as long as they don’t have any dependencies, so I’d go with whichever option shortens the data dependencies on the critical path. The computer will…

You're right that the addition vs. multiplication issue likely does not matter on a modern chip. I just gave the example because it shows how the CPU architecture can affect how you write the code. I do not recall precisely when or where I heard the idea, but it was about a decade ago --- ages ago by computing standards.

Re: The art of high performance computing

#19

Post by rramadass »

Just amazed at how the author has created (and shared for free) such a comprehensive set of books including teaching C++ and Unix tools! There is something to learn for all Programmers (HPC specific or not) here.

Related: Jorg Arndt's "Matters Computational" book and FXT library - https://www.jjj.de/fxt/

Re: The art of high performance computing

#20

Post by crabbone »

dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

You'd be surprised how actually backwards and primitive are the tools used in HPC.

Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech with crappy / nonexistent documentation and inept tech support.

The interface between WLMs and the user who wants to use some resources is through submitting "jobs". These jobs can be interactive, but most often they are the so-called "batch jobs". A batch job is usually defined as... a Unix Shell script, where the comments are parsed to interpret those as instructions to the WLM. In the world with dozens of configuration formats... they chose to do this: embed configuration into Shell comments.

Debugging job failures is a nightmare, mostly because WLM software has really poor quality of execution. Pathetic error reporting. Idiotic defaults. Everything is so fragile it falls apart if you just as much as look at it in the wrong way. Working with it reminds me the very early days of Linux, when sometimes things just won't build, or would segfault right after you've tried running them, and there wasn't much you could do beside spending days or weeks trying to debug it just to get some basic functionality going.

When I have to deal with it, I feel kind of like in a steam-punk movie. Some stuff is really advanced, and then you find out that this advanced stuff is propped by some DIY retro nonsense you thought have died off decades ago. The advanced stuff is usually more on the side of hardware, while software is not keeping up with it for the most part.

Post reply on HN