Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reflections on 30 years of HPC programming

chapel-lang.org

Post reply on HN
Print view View original

61–70 of 129 posts

Re: Reflections on 30 years of HPC programming

#61

Post by kmaitreys »

zozbot234 wrote: post #43
kmaitreys wrote: post #39

Earlier quoted context omitted.

Can you elaborate on this? Slightly concerned because I have written (and planning to write more) Rust HPC code

Stable Rust doesn't have a local allocator construct yet, you can only change the global allocator or use a separate crate to provide a local equivalent.

Right. I have seen Zig where one needs to specify allocators as well. I'm sorry I'm not well versed enough to know how it makes things better for HPC though?

For now my plan is to write fairly similar style code as one may write in C++/Fortran through MPI bindings in Rust.

Re: Reflections on 30 years of HPC programming

#62

Post by bluedino »

jpecar wrote: post #8

All these fancy HPC languages are all nice and dandy, but the hard reality I see on our cluster is that most of the work is done in Python, R and even Perl and awk. MPI barely reached us and people still prefer huge single machines to proper distributed computing. Yeah, bioinformatics is from another planet.

Python is huge in AI/ML etc as well.

I haven't talked to anyone writing C++ code on a HPC cluster that I'm working on in a long, long time. And that's in industrial/chemical/automotive fields.

Re: Reflections on 30 years of HPC programming

#63

Post by bluedino »

kinow wrote: post #4
riffraff wrote: post #2

Perhaps one issue lacking discussion in the article is how easy it is to find devs? I've never worked in HPC but it seems it should be relatively simple to find a C/C++ dev that can pick up OpenMP, or one that already knows it, compared to hiring people who know Chapel. The "scaling down" factor (how easy or interesting it is to use tool X for small use) seems a disadvantage of HPC-only languages, which creates a bar…

I think hpc devs need an extra set of skills that are not so common. Such as parallel file systems, batch schedulers, NUMA, infiniband, and probably some domain-specific knowledge for the apps they will develop. This knowledge is also probably a bit niche, like climate modelling, earthquake simulation, lidar data processing, and so it goes. And even knowing OpenMP or MPI may not suffice if the site uses older version…

Most developers have no clue about any of that stuff. It's all abstracted out.

Re: Reflections on 30 years of HPC programming

#64

Post by suuuuuuuu »

pjmlp wrote: post #57
suuuuuuuu wrote: post #54

Earlier quoted context omitted.

CUDA is not C++. CUDA for GPU kernels is its own language. That's the actual problem requiring new languages or abstractions.

Says those that don't know CUDA. You can program CUDA in standard C++20, with CUDA libraries hidding the language extensions. I love when C and C++ dialects are C and C++ when it matters, and not when it doesn't help to sell the ideas being portrayed.

Sorry, I wasn't aware of these developments (having abandoned CUDA for hardware-agnostic solutions before 2020). It doesn't change my point anyway, if it's specific to a single vendor.

I'm extremely dubious that such an opaque abstraction can actually solve the (true) problem. "Not having to write CUDA" is not enough - how do you tune performance? Parallelization strategies, memory prefetching and arrangement in on-chip caches, when to fuse kernels vs. not... I don't doubt the compiler can do these things, but I do doubt that it can know at compile time what variants of kernel transformations will optimize performance on any given hardware. That's the real problem: achieving an abstraction that still gives one enough control to achieve peak performance.

Edit: you tell me if I'm wrong, but it seems that std::par can't even use shared memory, let alone let one control its usage? If so, then my point stands: C++ is not remotely relevant. Again, avoiding writing CUDA (etc.) doesn't solve the real problem that high-performance language abstractions aim to address.

Re: Reflections on 30 years of HPC programming

#65

Post by m_mueller »

_flux wrote: post #29
lugu wrote: post #24

Earlier quoted context omitted.

Parent talks about new languages, as per the article Fortran or C doing fine. I speculate the benefit of C++ over Rust how it let programmers instruct the compiler of warranty that goes beyong the initial semantic of the language. See __restrict, __builtin_prefetch and __builtin_assume_aligned. The programming language is a space for conversations between compiler builders and hardware designers.

I believe __restrict, and __builtin_prefetch/__builtin_assume are compiler extensions, not part of the C++ language as is, and different compilers implement (or don't) these differently. The rust compiler actually has similar things, but they're not available in stable builds. I suppose there are some issues if principle why not to include them in stable. E.g: https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read…

btw. Fortran is implicitly behaving as "restrict" by default, which makes sense together with its intuitive "intent" system for function/subroutine arguments. This is one of the biggest reasons why it's still so popular in HPC - scientists can pretty much just write down their equations, follow a few simple rules (e.g. on storage order) and out comes fairly performant machine code. Doing the same (a 'naive' first implementation) in C or C++ usually leads to something severely degraded compared to the theoretical limits of a given algorithm on given hardware.

Re: Reflections on 30 years of HPC programming

#66

Post by bayindirh »

bruce343434 wrote: post #7
jandrewrogers wrote: post #3

I can easily explain this, having worked in this space. The new languages don’t actually solve any urgent problems. How people imagine scalable parallelism works and how it actually works doesn’t have a lot of overlap. The code is often boringly single-threaded because that is optimal for performance. The single biggest resource limit in most HPC code is memory bandwidth. If you are not addressing this then you are n…

What does it mean to be friendly to memory bandwidth, and why does C++ excel at it, over, say, Fortran or C or Rust?

Actually, C, FORTRAN and C++ are friendly to memory bandwidth, written correctly.

C++ is better than FORTRAN, because while it's being still developed and quite fast doing other things that core FORTRAN is good at is hard. At the end of the day, it computes and works well with MPI. That's mostly all.

C++ is better than C, because it can accommodate C code inside and has much more convenience functions and libraries around and modern C++ can be written more concisely than C, with minimal or no added overhead.

Also, all three languages are studied so well that advanced programmers can look a piece of code and say that "I can fix that into the cache, that'll work, that's fine".

"More modern" programming languages really solve no urgent problems in HPC space and current code works quite well there.

Reported from another HPC datacenter somewhere in the universe.

Re: Reflections on 30 years of HPC programming

#67

Post by bayindirh »

j4k0bfr wrote: post #9
jandrewrogers wrote: post #3

I can easily explain this, having worked in this space. The new languages don’t actually solve any urgent problems. How people imagine scalable parallelism works and how it actually works doesn’t have a lot of overlap. The code is often boringly single-threaded because that is optimal for performance. The single biggest resource limit in most HPC code is memory bandwidth. If you are not addressing this then you are n…

I'm pretty interested in realtime computing and didn't realise C++ was considered bandwidth efficient! Coming from C, I find myself avoiding most 'new' C++ features because I can't easily figure out how they allocate without grabbing a memory profiler.

You can always go through cachegrind or perf and see what happens with your code.

I managed to reach practical IPC limits of the hardware I was running on, and while I could theoretically make prefetcher happier with some matrix reordering, looking back, I'm not sure how much performance it provided since the FPU was already saturated at that point.

Re: Reflections on 30 years of HPC programming

#68

Post by calvinmorrison »

zozbot234 wrote: post #44
jltsiren wrote: post #14

Earlier quoted context omitted.

Bioinformatics is an outlier within HPC. It's less about numerical computing and more about processing string data with weird algorithms and data structures that are rarely used anywhere else. Distributed computing never really took off in bioinformatics, because most tasks are conveniently small. For example, a human genome is small enough that you can run most tasks involving a single genome on an average cost-effe…

Bioinformatics is also seeing huge gains from rewriting the slow Python code into highly parallel Rust (way less fiddly than C++ for the typical academic dev).

This is not new either. Most of numpy and pandas and other stuff you use the Python C interface and pass arrays in and get data back. You can write small embeddable C libraries pretty easily for real crunching and you get the ease of writing python (basically comprehensible to researchers who understand The MATLAB )

Re: Reflections on 30 years of HPC programming

#69

Post by convolvatron »

kmaitreys wrote: post #61
zozbot234 wrote: post #43

Earlier quoted context omitted.

Stable Rust doesn't have a local allocator construct yet, you can only change the global allocator or use a separate crate to provide a local equivalent.

Right. I have seen Zig where one needs to specify allocators as well. I'm sorry I'm not well versed enough to know how it makes things better for HPC though? For now my plan is to write fairly similar style code as one may write in C++/Fortran through MPI bindings in Rust.

if you're using thread level parallelism, there is always a benefit to having a per-thread allocator so that you don't have to take global locks to get memory, they become highly contended.

if you take that one step further and only use those objects on a single core, now your default model is lock-free non-shared objects. at large scale that becomes kind of mandatory. some large shared memory machines even forgo cache consistency because you really can't do it effectively at large scale anyways.

but all of this is highly platform dependent, and I wouldn't get too wrapped up around it to begin with. I would encourage you though to worry first about expressing your domain semantics, with the understanding that some refactoring for performance will likely be necessary.

if you have the patience and personally and within the project, it can be a lot of fun to really get in there and think about the necessary dependencies and how they can be expressed on the hardware. there's a lot of cool tricks, for example trading off redundant computation to reduce the frequency of communication.

Re: Reflections on 30 years of HPC programming

#70

Post by _flux »

m_mueller wrote: post #65
_flux wrote: post #29

Earlier quoted context omitted.

I believe __restrict, and __builtin_prefetch/__builtin_assume are compiler extensions, not part of the C++ language as is, and different compilers implement (or don't) these differently. The rust compiler actually has similar things, but they're not available in stable builds. I suppose there are some issues if principle why not to include them in stable. E.g: https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read…

btw. Fortran is implicitly behaving as "restrict" by default, which makes sense together with its intuitive "intent" system for function/subroutine arguments. This is one of the biggest reasons why it's still so popular in HPC - scientists can pretty much just write down their equations, follow a few simple rules (e.g. on storage order) and out comes fairly performant machine code. Doing the same (a 'naive' first imp…

Oh I actually had some editing mistake, I meant to say that also Rust has restrict by default, by virtue of all references being unique xor readonly.

As I understand it, the Fortran compiler just expects your code to respect the "restrictness", it doesn't enforce it.

Post reply on HN