Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The art of high performance computing

theartofhpc.com

Post reply on HN
Print view View original

41–50 of 125 posts

Re: The art of high performance computing

#41

Post by keefle »

CoastalCoder wrote: post #28
rlupi wrote: post #24

I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…

This seemed like a big topic when I was interviewing with Meta and nVidia some months ago. Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.

Could you link me the YouTube videos/articles in question? It happens to be my research area and I'm interested in knowing how big companies such as meta deal with multi-GPU systems

Re: The art of high performance computing

#42

Post by victotronics »

bee_rider wrote: post #12
mkoubaa wrote: post #2

UT Austin really is a fantastic institution for HPC and computational methods.

Every BLAS you want to use has at least some connection to UT Austin’s TACC.

Not quite. Every modern BLAS is (likely) based on Kazushige Goto's implementation, and he was indeed at TACC for a while. But probably the best open source implementation "BLIS" is from UT Austin, but not connected to TACC.

Re: The art of high performance computing

#43

Post by throwawaaarrgh »

bluedino wrote: post #37
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any k…

I always found that funny too. A business who needs a powerful computing solution can come up with some amazingly robust stuff, whereas science/research just buys a big mainframe and hopes it works.

Re: The art of high performance computing

#44

Post by bee_rider »

victotronics wrote: post #42
bee_rider wrote: post #12

Earlier quoted context omitted.

Every BLAS you want to use has at least some connection to UT Austin’s TACC.

Not quite. Every modern BLAS is (likely) based on Kazushige Goto's implementation, and he was indeed at TACC for a while. But probably the best open source implementation "BLIS" is from UT Austin, but not connected to TACC.

Oh really? I thought BLIS was from TACC. Oops, mea culpa.

Re: The art of high performance computing

#45

Post by CoastalCoder »

keefle wrote: post #41
CoastalCoder wrote: post #28

Earlier quoted context omitted.

This seemed like a big topic when I was interviewing with Meta and nVidia some months ago. Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.

Could you link me the YouTube videos/articles in question? It happens to be my research area and I'm interested in knowing how big companies such as meta deal with multi-GPU systems

I don't have them bookmarked anymore, but they may have been from this playlist: [0]

[0] https://www.youtube.com/playlist?list=PLBnLThDtSXOw_kePWy3CS...

Re: The art of high performance computing

#46

Post by timoteostewart »

synergy20 wrote: post #27
rlupi wrote: post #24

I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…

check out openbmc project and DTMF association

DMTF (not DTMF)

https://www.dmtf.org/

Re: The art of high performance computing

#47

Post by RhysU »

bee_rider wrote: post #44
victotronics wrote: post #42

Earlier quoted context omitted.

Not quite. Every modern BLAS is (likely) based on Kazushige Goto's implementation, and he was indeed at TACC for a while. But probably the best open source implementation "BLIS" is from UT Austin, but not connected to TACC.

Oh really? I thought BLIS was from TACC. Oops, mea culpa.

https://github.com/flame/blis/

Field et al, recent winners of the James H. Wilkinson Prize for Numerical Software.

Field and Goto both collaborated with Robert van de Geijn. Lots of TACC interaction in that broader team.

Re: The art of high performance computing

#48

Post by LASR »

The hardware / datacenter side of this is equally fascinating.

I used to work in AWS, but on the software / services side of things. But now and then, we would crash some talks from the datacenter folks.

One key relevation for me was that increasing compute power in DCs is primarily a thermodynamics problem than actual computing. The nodes have become so dense that shipping power in and shipping heat out, with all kinds of redundancies is an extremely hard problem. And it's not like you can perform a software update if you've discovered some inefficiencies.

This was ~10 years ago, so probably some things have changed.

What blows me away is that Amazon, starting out as an internet bookstore is at the cutting edge of solving thermodynamics problems.

Re: The art of high performance computing

#49

Post by justin66 »

bluedino wrote: post #37
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any k…

> The one drawback I find is that a lot of HPC jobs want you do have a masters degree.

Is it possible that pretty much any specialization, outside of the most common ones, engages in a lot of gatekeeping? I remember how difficult it appeared to be after I graduated to break into embedded systems (I never did). I persisted until I realized it doesn't even pay very well, comparatively.

Re: The art of high performance computing

#50

Post by cavisne »

rlupi wrote: post #24

I am interested in the more hardware management side of HPC (how problems are detected, diagnosed, mapped into actions such as reboot/reinstall/repairs, how these are scheduled and how that is optimized to provide the best level of service, how this is done if there are multiple objectives to optimize at once e.g. node availability vs overall throughput, how different topologies affect the above, how other constraint…

This paper from Microsoft [1] is the coolest thing I've seen in this space. Basically workload (deep learning in this case) level optimization to allow jobs to be resized and preempted.

[1] https://arxiv.org/pdf/2202.07848.pdf

Post reply on HN