Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reflections on 30 years of HPC programming

chapel-lang.org

Post reply on HN
Print view View original

71–80 of 129 posts

Re: Reflections on 30 years of HPC programming

#71

Post by hpcdude »

HPCdude here, and this is a mostly correct article, but here are what it misses:

1) It mentions in passing the hardware abstraction not being as universal as it seemed. This is more and more true, once we started doing fpgas, then asics, and as ARM and other platforms starting making headway, it fractured things a bit.

GPUs too: I'm still a bit upset about CUDA winning over OpenCL, but Vulkan compute gives me hope. I haven't messed with SYCL but it might be a future possibility too.

2) The real crux is the practical, production interfaces that HPC end users get. Normally I'm not exposing an entire cluster (or sub cluster) to a researcher. I give them predefined tools which handle the computation across nodes for them (SLURM is old but still huge in the HPC space for a reason! When I searched the article for "slurm" I got 0 hits!) When it comes to science, reproducibility is the name of the game, and having more idempotent and repeatable code structures are what help us gain real insights that others can verify or take and run with into new problem/solutions. Ad-hoc HPC programming doesn't do that well, like existing languages slurm and other orchestration layers handle.

Sidenote: One of the biggest advances recently is in RDMA improvements (remote direct memory access), because the RAM needs of these datasets are growing to crazy numbers, and often you have nodes being underutilized that are happy to help. I've only done RoCE myself though and not much with Infiniband, (sorry Yale, thats why I flubbed on the interview) but honestly, I still really like RoCE for cluster side and LACP for front facing ingress/egress.

The point is existing tooling can be massaged and often we don't need new languages. I did some work with Mellanox/Weka prior to them being bought by Nvidia on optimizing the kernel shims for NFSv4 for example. Old tech made fast again.

Re: Reflections on 30 years of HPC programming

#72

Post by moregrist »

_flux wrote: post #29
lugu wrote: post #24

Earlier quoted context omitted.

Parent talks about new languages, as per the article Fortran or C doing fine. I speculate the benefit of C++ over Rust how it let programmers instruct the compiler of warranty that goes beyong the initial semantic of the language. See __restrict, __builtin_prefetch and __builtin_assume_aligned. The programming language is a space for conversations between compiler builders and hardware designers.

I believe __restrict, and __builtin_prefetch/__builtin_assume are compiler extensions, not part of the C++ language as is, and different compilers implement (or don't) these differently. The rust compiler actually has similar things, but they're not available in stable builds. I suppose there are some issues if principle why not to include them in stable. E.g: https://doc.rust-lang.org/std/intrinsics/fn.prefetch_read…

restrict is in C99. I’m not sure why standard C++ never adopted it, but I can guess: it can be hard to reason about two restrict’d pointers in C, and it probably becomes impossible when it interacts with other C++ features.

The rest are compiler extensions, but if you’re in the space you quickly learn that portability is valued far less than program optimization. Most of the point of your large calculations is the actual results themselves, not the code that got you there. The code needs to be correct and reproducible, but HPC folks (and grant funding agencies) don’t care if your Linux/amd64 program will run, unported, on Windows or on arm64. Or whether you’ve spent time making your kernels work with both rocm and cuda.

Re: Reflections on 30 years of HPC programming

#73

Post by pphysch »

pama wrote: post #30
crabbone wrote: post #18

As someone who worked for a while and still works in HPC, my impression from this field as compared to eg. programming in finance sector or programming for storage sector is that... HPC is so backwards and far behind, it's really amazing how it's portrayed as some sort of a champion of the field. That's not to say that new things don't happen there, it's just that I find a lot of old stuff that was shown to be bad de…

HPCs never loved the inefficiencies of anything virtualized (VMs or any containers really), so the shell hacks of module enabled a (limited, but workable) level of reproducibility that was sufficiently composable and usable by researchers who understood the shell. I am not going to defend this tcl hack any further, but I can see how it was the path of least resistance when people tried to stay close to the raw metal…

I promise you that the main reason HPC is behind on virtualization is not because of the little bit of overhead. There are a dozen other inefficiencies in the average HPC workload that are more significant.

Most centers don't even have good real-time observability systems to diagnose systemic inefficiencies, leaving application/workload profiling purely up to user-space.

The HP in HPC has really been watered down over the last couple decades, and "IT for computational research" would be a more accurate name. You can do genuinely high-performance computing there, but you'll be an outlier.

Re: Reflections on 30 years of HPC programming

#74

Post by SiempreViernes »

pjmlp wrote: post #56
RhysU wrote: post #47

Earlier quoted context omitted.

The author's framing "we have failed" suggests otherwise. This section, https://chapel-lang.org/blog/posts/30years/#ok-then-why , does not mention libraries at all.

Not really, you should actually read that section a few times as well. > A fact of life in HPC is that the community has many large, long-lived codes written in languages like Fortran, C, and C++ that remain important. Such codes keep those languages at the forefront of peoples’ minds and sometimes lead to the belief that we can’t adopt new languages. > In large part because of the previous point, our programming not…

The fact that the author is a developer of Chapel pretty neatly explains why "no new language was adopted" is valued as failure, the article itself makes little effort to argue for that value judgment.

Re: Reflections on 30 years of HPC programming

#75

Post by jacquesm »

Joel_Mckay wrote: post #59
jacquesm wrote: post #50

Earlier quoted context omitted.

Yes, it's absolutely amazing. But I'm intimately familiar with how the Erlang VM works and it would seem to me to be a very bad match for HPC on the number crunching side, though it likely would do quite well on the orchestration side but that would require the people writing the rest of the code to change their way of working completely. And given how much of that is still F77 I highly doubt they would be willing to…

For simple AMQP, it has performed rather well for our use-cases over the years. Haven't personally deployed this version yet. ymmv =3 https://www.rabbitmq.com/docs/quorum-queues

Of course it does, that's an ideal usecase. But the topic is a different kettle of fish.

And I'm saying that as a complete fan of Erlang, it is one of the few pieces of software 'done right' that I'm aware of. Unfortunately that comes at the price of being unsuitable for highly optimized number crunching unless you want to break out the C-compiler and write an extension.

Python is similar, but there the extensions have been grafted on so well lots of people forget that they are not part of the language itself. In the Erlang world you'd have a lot of leaks and conversions to make something like that work and it would likely never be as transparent as python, which in many ways is both the new BASIC and the new FORTRAN.

Re: Reflections on 30 years of HPC programming

#76

Post by kmaitreys »

convolvatron wrote: post #69
kmaitreys wrote: post #61

Earlier quoted context omitted.

Right. I have seen Zig where one needs to specify allocators as well. I'm sorry I'm not well versed enough to know how it makes things better for HPC though? For now my plan is to write fairly similar style code as one may write in C++/Fortran through MPI bindings in Rust.

if you're using thread level parallelism, there is always a benefit to having a per-thread allocator so that you don't have to take global locks to get memory, they become highly contended. if you take that one step further and only use those objects on a single core, now your default model is lock-free non-shared objects. at large scale that becomes kind of mandatory. some large shared memory machines even forgo cac…

Thank you for such a great reply!

There's a lot of useful advice here that'll surely come in handy to me later. For now, yeah I'm just going to try to make things work. So far I have mostly written intra-node code for which rayon has been adequate. I haven't gotten around to test the ergonomics of rs-mpi. But it feels quite an exciting prospect for sure.

Re: Reflections on 30 years of HPC programming

#77

Post by sliken »

crabbone wrote: post #18

As someone who worked for a while and still works in HPC, my impression from this field as compared to eg. programming in finance sector or programming for storage sector is that... HPC is so backwards and far behind, it's really amazing how it's portrayed as some sort of a champion of the field. That's not to say that new things don't happen there, it's just that I find a lot of old stuff that was shown to be bad de…

As you dig deeper I think you'll find a method behind the madness.

Sure modules just play with env variables. But it's easy to inspect (module show), easy to document "use modules load ...", allows admins to change the default when things improve/bug fixed, but also allows users to pin the version. It's very transparent, very discover-able, and very "stale". Research needs dictate that you can reproduce research from years past. It's much easier to look at your output file and see the exact version of compiler, MPI stack, libraries, and application than trying to dig into a container build file or similar. Not to mention it's crazy more efficient to look at a few lines of output than to keep the container around.

As for slurm, I find it quite useful. Your main complaint is no default systemd service files? Not like it's hard to setup systemd and dependencies. Slurms job is scheduling, which involves matching job requests for resources, deciding who to run, and where to run it. It does that well and runs jobs efficiently. Cgroup v2, pinning tasks to the CPU it needs, placing jobs on CPU closest to the GPU it's using, etc. When combined with PMIX2 it allows impressive launch speeds across large clusters. I guess if your biggest complaint is the systemd service files that's actually high praise. You did mention logging, I find it pretty good, you can increase the verbosity and focus on server (slurmctld) or client side (slurmd) and enable turning on just what you are interested, like say +backfill. I've gotten pretty deep into the weeds and basically everything slurm does can be logged, if you ask for it.

Sounds like you've used some poorly run clusters, I don't doubt it, but I wouldn't assume that's HPC in general. I've built HPC clusters and did not use the university's AD, specifically because it wasn't reliable enough. IMO a cluster should continue to schedule and run jobs, even if the uplink is down. Running a past EoL OS on an HPC cluster is definitely a sign that it's not run well and seems common when a heroic student ends up managing a cluster and then graduates leaving the cluster unmanaged. Sadly it's pretty common for IT to run a HPC cluster poorly, it's really a different set of contraints, thus the need for a HPC group.

Plenty of HPC clusters out there a happy to support the tools that helps their users get the most research done.

Re: Reflections on 30 years of HPC programming

#78

Post by convolvatron »

jandrewrogers wrote: post #3

I can easily explain this, having worked in this space. The new languages don’t actually solve any urgent problems. How people imagine scalable parallelism works and how it actually works doesn’t have a lot of overlap. The code is often boringly single-threaded because that is optimal for performance. The single biggest resource limit in most HPC code is memory bandwidth. If you are not addressing this then you are n…

I worked in parallel computing in the late 80s and early 90s when parallel languages were really a thing. in HPC applications memory bandwidth is certainly a concern, although usually the global communications bandwidth (assuming they are different) is the roofline. by saying c++ you're implying that MPI is really sufficient, and its certainly possible to prop up parallel codes with MPI is really quite tiresome and hard to play with the really interesting problem which is the mapping of the domain state across the entire machine.

other hugely important problems that c++ doesn't address are latency hiding, which avoids stalling out your entire core waiting for distributed message, and a related solution which is interleave of computation and communication.

another related problem is that a lot of the very interesting hardware that might exist to do things like RDMA or in-network collective operations or even memory-controller based rich atomics, aren't part of the compiler's view and thus are usually library implementations or really hacky inlines.

is there a good turnkey parallel language? no. is there sufficient commonality in architecture or even a lot of investment in interesting ideas that were abandoned because of cost, no. but there remains a huge potential to exploit parallel hardware with implicit abstractions, and I think saying 'just use c++' is really missing almost all of the picture here.

addendum: even if you are working on a single-die multicore machine, if you don't account for locality, it doesn't matter how good your code generator is, you will saturate the memory network. so locality is an important and languages like Chapel are explicitly trying to provide useful abstractions for you to manage it.

Re: Reflections on 30 years of HPC programming

#79

Post by RandomTeaParty »

Joel_Mckay wrote: post #20
jacquesm wrote: post #16

Earlier quoted context omitted.

That will never happen. The overhead is massive.

People did try to create an OTP in HDL at one point. And Erlang has already run many telecom infrastructures for decades. Surprising given how fragile the multi-host implementation has proven. Erlang/Elixir are neat languages, and right next to Julia for fun. =3

Erlang is about reliability, hpc is about performance (literally in the name)

Re: Reflections on 30 years of HPC programming

#80

Post by pjmlp »

SiempreViernes wrote: post #74
pjmlp wrote: post #56

Earlier quoted context omitted.

Not really, you should actually read that section a few times as well. > A fact of life in HPC is that the community has many large, long-lived codes written in languages like Fortran, C, and C++ that remain important. Such codes keep those languages at the forefront of peoples’ minds and sometimes lead to the belief that we can’t adopt new languages. > In large part because of the previous point, our programming not…

The fact that the author is a developer of Chapel pretty neatly explains why "no new language was adopted" is valued as failure, the article itself makes little effort to argue for that value judgment.

> I didn’t put Chapel on my list of broadly adopted HPC programming notations above, in large part to avoid being presumptuous. But it’s also because, regrettably, I don’t consider Chapel’s support within the community to be as solid as the others on my list
Post reply on HN