The art of high performance computing
theartofhpc.com
The art of high performance computing
1–10 of 125 posts
Re: The art of high performance computing
#2UT Austin really is a fantastic institution for HPC and computational methods.
Re: The art of high performance computing
#3I was asked to share a TA role on a graduate course in HPC a decade ago. I turned down the offer.
After a cursory glance, I can honestly say that if this book were available then, I'd have taken the opportunity.
The combination of what I perceive to be Knuth's framing of art, along with carpentry and the need to be a better devops person than your devops person is compelling.
Kudos to the author for such an achievement. UT Austin seems to have achieved in computer science what North Texas State did in music.
Re: The art of high performance computing
#4[deleted]
Re: The art of high performance computing
#5I took a course on scientific computing in 2013. It was cross-listed under both the computer science and applied math departments. The issue is that the field is pretty broad overall and a lot of topics were covered in a cursory manner, including anything related to HPC and parallel programming in particular. I don't regret taking the course, but it was too broad for the applications I was pursuing.
I haven't looked at what courses are being offered in several years, but when I was a graduate student, I really would have benefited from a dedicated semester-long course on parallel computing, especially going into the weeds about particular algorithms and data structures in parallel and distributed computing. Those were handled in a super cursory manner in the scientific computing course I took, as if somehow you'd know precisely how to parallelize things the first time you try. I've since learned a lot of this stuff on my own and from colleagues over the years, as many people do in HPC, but books like these would have been invaluable as part of a dedicated semester-long course.
Re: The art of high performance computing
#6It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worry about these details? I'm also curious if HPC practitioners have to fiddle a lot of black-box knobs to squeeze out performance?
Re: The art of high performance computing
#7Is there something wrong with the GitHub files since I cannot render any of the textbooks PDF files?
https://github.com/VictorEijkhout/TheArtofHPC_pdfs/blob/main...
Re: The art of high performance computing
#8Is there something wrong with the GitHub files since I cannot render any of the textbooks PDF files? https://github.com/VictorEijkhout/TheArtofHPC_pdfs/blob/main...
I think the files are too large to render in the github browser and they give an error. You can pick the 'download raw' option to download locally and read the file. Worked for me.
Re: The art of high performance computing
#9It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
I don’t think I do HPC (I only will use up to, say, 8 nodes at a time), but the impression I get is that they are already working on quite hard problems at the high-level, so they need to lean on good libraries for the low-level stuff, otherwise it is just too much.
Re: The art of high performance computing
#10It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…
Yes and no.
MPI and OpenMP are the primary abstractions from the hardware in HPC, with MPI being an abstracted form of distributed-memory parallel computing and OpenMP being an abstracted form of shared-memory parallel computing. Many researchers write their codes purely using those, often both in the same code. When using those, you really do not need to worry about the architectural details most of the time.
Still, some researchers who like to further optimize things do in fact fiddle with a lot of small architectural details to increase performance further. For example, loop unrolling is pretty common and can get quite confusing in my opinion. I vaguely recall some stuff about trying to vectorize operations by preferring addition over multiplication due to the particular CPU architecture, but I do not think I've seen that in practice.
Preventing cache misses is another major one, where some codes are written so that the most needed information is stored in the CPU's cache rather than memory. Most codes only handle this by ensuring column-major order loops for array operations in Fortran or row-major order loops in C, but the concept can be extended further. If you know the cache size for your processors, you could hypothetically optimize some operations to keep all of the needed information inside the cache to minimize cache misses. I've never seen this in practice but it was actively discussed in the scientific computing course I took in 2013.
The use of particular GPUs depends heavily on the problem being solved, with some being great on GPUs and others being too difficult. I'm not too knowledgeable about that, unfortunately.