Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reflections on 30 years of HPC programming

chapel-lang.org

Post reply on HN
Print view View original

111–120 of 129 posts

Re: Reflections on 30 years of HPC programming

#111

Post by pphysch »

saltcured wrote: post #109
pphysch wrote: post #73

Earlier quoted context omitted.

I promise you that the main reason HPC is behind on virtualization is not because of the little bit of overhead. There are a dozen other inefficiencies in the average HPC workload that are more significant. Most centers don't even have good real-time observability systems to diagnose systemic inefficiencies, leaving application/workload profiling purely up to user-space. The HP in HPC has really been watered down ove…

It's a mixture of legacy and reality. For one, the assumption has been that you had dedicated use of all the nodes and communication network. It would kill your performance if your local node CPU scheduler was interfering with having your actual HPC program active when the messages were coming in from its peer tasks on the other nodes, since parallel jobs are limited in the end by the critical path latency of the cro…

Agreed on all points and this captures the history well.

Re: Reflections on 30 years of HPC programming

#112

Post by nine_k »

bradcray wrote: post #102
nine_k wrote: post #101

Earlier quoted context omitted.

I suppose that most HPC problems are embarrassingly parallel™, and have very little if any mutable shared state?

I'd say that the opposite is more often the reality, which is why HPC systems tend to have high-bandwidth, low-latency networks.

High bandwidth may mean the need to consult some very large but immutable data structure. As a trivial example, multiplying two matrices requires accessing each matrix fully multiple times over, but neither of them is altered in the process, so it can safely be done in parallel. Recording the result of a (naive) matrix multiplication can also be done without programmatic coordination, because each element is only updated once, independently from others.

This is very unlike, say, a database engine, where mutations occur all the time and may come from multiple threads.

Rust specifically makes it hard to impossible to clobber shared mutable state, e.g. to produce a dangling pointer. But this is not a problem that our matrix-multiplication example would have, so it won't benefit from being implemented in Rust. Maybe this applies to more classes of HPC problems.

Re: Reflections on 30 years of HPC programming

#113

Post by godelski »

nine_k wrote: post #112
bradcray wrote: post #102

Earlier quoted context omitted.

I'd say that the opposite is more often the reality, which is why HPC systems tend to have high-bandwidth, low-latency networks.

High bandwidth may mean the need to consult some very large but immutable data structure. As a trivial example, multiplying two matrices requires accessing each matrix fully multiple times over, but neither of them is altered in the process, so it can safely be done in parallel. Recording the result of a (naive) matrix multiplication can also be done without programmatic coordination, because each element is only upd…

The HPC infrastructure is not like you're used to using. It is very high bandwidth but latency is dependent on where your data lives. There's a lot more layers that complicate things and each layer has a very different I/O speed

https://extremecomputingtraining.anl.gov/sites/atpesc/files/...

Also how to handle the data can be very different. Just see how libraries like this work. They take advantage of those burst buffers and try to minimize what's being pulled from storage. Though there's a lot of memory management in the code people write to do all this complex stuff you need so that you aren't waiting around for disks... or worse... tape

https://adios-io.org/applications/

Re: Reflections on 30 years of HPC programming

#114

Post by pjmlp »

bjourne wrote: post #110
pjmlp wrote: post #57

Earlier quoted context omitted.

Says those that don't know CUDA. You can program CUDA in standard C++20, with CUDA libraries hidding the language extensions. I love when C and C++ dialects are C and C++ when it matters, and not when it doesn't help to sell the ideas being portrayed.

If CUDA is C++ then I'd like to know how you throw and catch exceptions in CUDA kernels.

The same way as people writing C++ code as Google employees do, including LLVM and Chrome.

Re: Reflections on 30 years of HPC programming

#116

Post by uecker »

grg0 wrote: post #98
bruce343434 wrote: post #7

Earlier quoted context omitted.

What does it mean to be friendly to memory bandwidth, and why does C++ excel at it, over, say, Fortran or C or Rust?

I'd say it's being able to structure your data however suits your problem and your hardware, then being able to look at a profile and being able to map read/writes back to source. Both C and C++ excel at this. The advantage of C++ over C is that, with care, you can write zero-cost abstractions over whatever mess your data ends up as, and make the API still look intuitive. C isn't as good here.

According to my experience, all the "zero-cost abstractions" from C++ most of the time make it more annoying to maintain and/or understand the code, especially with respect to resource management, introduce compatibility issues at the toolchain level, and - even when looking perfect in toy benchmarks - are often not even zero-cost (e.g. all the bloat the templates generate often hurts).

Re: Reflections on 30 years of HPC programming

#117

Post by uecker »

calvinmorrison wrote: post #68
zozbot234 wrote: post #44

Earlier quoted context omitted.

Bioinformatics is also seeing huge gains from rewriting the slow Python code into highly parallel Rust (way less fiddly than C++ for the typical academic dev).

This is not new either. Most of numpy and pandas and other stuff you use the Python C interface and pass arrays in and get data back. You can write small embeddable C libraries pretty easily for real crunching and you get the ease of writing python (basically comprehensible to researchers who understand The MATLAB )

I would say the least thing the scientific community needs is the packaging mess of Python introduced also on the lower level via Rust.

Re: Reflections on 30 years of HPC programming

#118

Post by pjmlp »

rirze wrote: post #93
ivell wrote: post #38

I think Mojo has a good chance to become suitable for HPC.

I don't understand why people are so excited for Mojo. I don't get the impression it will replace anything but computational scripting that was done previously in Python. HPC is a different beast as far as I'm aware.

Additionally, as Intel, AMD and NVidia embrace Python JITs for their GPGPU, across all three major desktop OSes, the Mojo value becomes devalued.

Why rewrite in Mojo, if Intel, AMD and NVidia SDKs give first class support for Python, including writing kernels in a language subset.

Re: Reflections on 30 years of HPC programming

#119

Post by swiftcoder »

RandomTeaParty wrote: post #79
Joel_Mckay wrote: post #20

Earlier quoted context omitted.

People did try to create an OTP in HDL at one point. And Erlang has already run many telecom infrastructures for decades. Surprising given how fragile the multi-host implementation has proven. Erlang/Elixir are neat languages, and right next to Julia for fun. =3

Erlang is about reliability, hpc is about performance (literally in the name)

Erlang is also about orchestration - which HPC tends to need a fair bit of once you are out of the realm of embarrassingly parallel problems

Re: Reflections on 30 years of HPC programming

#120

Post by RhysU »

bradcray wrote: post #84
RhysU wrote: post #25

> we have failed to broadly adopt any new compiled programming languages for HPC The article neglects that all of C, C++, and Fortran have evolved over the last 30 years. Also, you'll find significant advances in the HPC library ecosystem over the trailing years. Consider, for example, Trilinos ( https://trilinos.github.io/index.html ) or Dakota ( https://dakota.sandia.gov/about-dakota/ ) both of which push a ton of…

The evolution of C, C++, and Fortran is touched on in a sidebar, although admittedly very briefly: > Champions of Fortran, C++, MPI, or other entries on this list could argue that…

Thank you for pointing out the sidebar. I missed this because I did not understand that footnotes/sidebars were presented as links.
Post reply on HN