Reflections on 30 years of HPC programming

But, when I encountered OpenCL and CUDA about ten years ago, I was struck by just how much these were delivering the SPMD parallel programming model in finished products. Around 1995, these were often C dialects with some wonky compiler that each research group just barely kept together. By 2015, they were just bundled up inside a graphics driver or similarly commoditized runtime environment.

Also, the GPU of 2015 was delivering the throughput we dreamed of in supercomputers back then. A teraFLOP went from a strategic theme to something you could deploy to your desktop.

Productivity: For me, while Fortran has some nice features for HPC (multidimensional arrays), lots about its design feels very old-fashioned to my (not particularly young) eyes. C and C++ are more "my generation" of programming language, so are familiar and comfortable, yet they still seem verbose, convoluted, and less readable (more symbolically oriented) as compared to Python, Julia, or Swift, which are more what I'm looking for in terms of productivity these days. Of the three, C++ has clearly made the biggest strides in recent years to improve productivity, with some successes in my opinion, though I've also had a hard time keeping up with all the changes.

Safety: I consider C and C++ to be fairly unsafe languages compared to more modern alternatives. I don't have enough experience with Fortran to have a particularly informed opinion, but feel as though I've been aware of patterns in the past that have felt unsafe. Here again, I think using modern C++ in a certain style (e.g., smart pointers) probably makes nice strides w.r.t. safety, but I'd still consider there to be a gap between it and Python/Rust (as does my colleague in this post: https://chapel-lang.org/blog/posts/memory-safety/)

Portability: Modulo the degree to which various compilers keep up with the latest standards in Fortran and C++, I'd consider all three languages to be quite portable.

Performance: There's no question that these are high-performing languages in the sequential computing setting. In HPC, while Fortran or C++ and MPI are often considered the gold standard, it's a standard that can be beat if your language maps more natively to the network's capabilities, or knows how to optimize for distributed memory computing rather than relying on the programmer to do it themselves.

With respect to the "10 myths" series, while the focus of the series was about combatting prevalent negative attitudes about new languages in the HPC community, I think there's a lot of content along the way that rationalizes the value of creating new languages in my rebuttals. That said, I fully realize that it's a long read, particularly in its updated "Redux" form.

Thanks again for clarifying your previous point.

For one, the assumption has been that you had dedicated use of all the nodes and communication network. It would kill your performance if your local node CPU scheduler was interfering with having your actual HPC program active when the messages were coming in from its peer tasks on the other nodes, since parallel jobs are limited in the end by the critical path latency of the cross-node communications.

It's only on the most "embarrassingly parallel" end of the spectrum where you can tolerate a bunch of virtualization and non-determinism, because the tasks communicate so infrequently or via such asynchronous mechanisms that they don't really impact the throughput of the whole job if they are asleep at random times.

But HPC systems also were very "unique". It wasn't just all Linux but a dozen different vendors' Unix variants with very different personalities. And for the bleeding-edge systems, each deployment was practically its own dialect of that vendor OS. Running a job was like cross-compiling to a one of a kind target. There was no generic platform where you could expect to build an app once and ship it around to whichever supercomputer was available.