Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The art of high performance computing

theartofhpc.com

Post reply on HN
Print view View original

111–120 of 125 posts

Re: The art of high performance computing

#111

Post by skyde »

dahart wrote: post #58
squidgyhead wrote: post #51

Earlier quoted context omitted.

Could you explain how you use a shmoo plot for optimization? Do you just have a performance metric at each point in parameter space?

The shmoo plot is just the name for measuring something (such as perf) over a range of parameter space. The simplest and most straightforward application is to pick a parameter or two that you don’t know what value they should be using, do the shmoo over the range of parameter space, and then set the knobs at whatever values give you the optimal measurement. Usually though, you have to iterate. Doing shmoos along the…

Bayesian optimization is very good an optimizing black box Knob in shmoo plot usually called meta parameter. Assuming each probing is costly they allow you to find the optional combination of value for many knob (dimension) with a minimum of probing

Re: The art of high performance computing

#112

Post by skyde »

bayindirh wrote: post #53
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

HPC admin here, generally serving "long tail of science" researchers. In today's x86_64 hardware, there's no "supercomputer memory subsystem". It's just a glorified NUMA system, and the biggest problem is putting the memory close to your core, i.e. keeping data local in your NUMA node to reduce latencies. Your resource mapping is handled by your scheduler. It knows your hardware, hence it creates a cgroup which satis…

You mentioned group I am very curious what you think of the work Facebook did on memory pressure metric

Re: The art of high performance computing

#113

Post by _naj »

crabbone wrote: post #20
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

You'd be surprised how actually backwards and primitive are the tools used in HPC. Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech wit…

Well, working in HPC, in my view the reason is the users (researchers/scientists) just want to use the machine as they have used it in the past, and this has been going on for many years now. That makes sense, as it allows them to focus on the scientific questions, not on IT stuff. They keep on using the tooling with which the software package they use came, and in some cases that is already 30 years old.

Re: The art of high performance computing

#114

Post by _naj »

bayindirh wrote: post #53
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

HPC admin here, generally serving "long tail of science" researchers. In today's x86_64 hardware, there's no "supercomputer memory subsystem". It's just a glorified NUMA system, and the biggest problem is putting the memory close to your core, i.e. keeping data local in your NUMA node to reduce latencies. Your resource mapping is handled by your scheduler. It knows your hardware, hence it creates a cgroup which satis…

I would contest the remark on Infiniband, since there is Slingshot which will be faster for real-world applications, and also Omnipath.

Re: The art of high performance computing

#115

Post by od0 »

bluedino wrote: post #37
dist1ll wrote: post #6

It's very interesting how abtracted away HPC sometimes looks from hardware. The books seem to revolve a lot around SPMD programming, algo & DS, task parallelism, synchronization etc, but very little about computer architecture details like supercomputer memory subsystems, high-bandwidth interconnects like CXL, GPU architecture and so on. Are the abstractions and tooling already good enough that you don't need to worr…

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any k…

would love to connect and talk more about HPC - let me know if you'd be up for a chat :)

Re: The art of high performance computing

#117

Post by bluedino »

od0 wrote: post #115
bluedino wrote: post #37

Earlier quoted context omitted.

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any k…

would love to connect and talk more about HPC - let me know if you'd be up for a chat :)

Emails in my profile, feel free to reach out

Re: The art of high performance computing

#118

Post by _a_a_a_ »

amluto wrote: post #105
_a_a_a_ wrote: post #100

Earlier quoted context omitted.

Of course you're not convecting but if you are radiating from a hot body into an ambient two Kelvin then you are going to lose heat really, really fast. IIRC heat loss by black body radiation into its surroundings is proportional to the fourth power of the temperature difference between the body and surroundings (from memory, and going back a very long way, so maybe incorrect).

> proportional to the fourth power of the temperature difference between the body and surroundings Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is on…

Thanks, you clearly know your subject. I must say that without doubting your figures, I am astonished at how little heat can be lost radiatively. Given the 4th powers I assumed it must be vastly higher, but clearly not, much against my intuition. Thanks for a good answer!

Re: The art of high performance computing

#119

Post by wickberg »

guenthert wrote: post #93

When joining a small company supporting the engineers of the HPC of a large car manufacturer, I was surprised to see so many in-house developed scripts around the scheduler (LSF). Only much later, when playing myself on a private miniature cluster with SLURM, I noticed that different versions of the scheduler software were generally incompatible to each other, i.e. one couldn't use one inside the cluster and another…

Cluster authentication actually remains the sore point for all interactions - for most systems the least-common denominator remains whether one can SSH into a login node. At which point the main job commands - sbatch/squeue - are pretty stable.

There have been some attempts to standardize basic job management APIs in the past - DRMAA being one noteworthy example. Although DRMAA v2 was only ever implemented by Grid Engine, and is effectively an lightly-abstracted version of their internal APIs, that has never really seen first-class adoption by Slurm/PBS/LSF.

For Slurm, the REST API is meant to be the way forward. It punts the authentication problem to, potentially, anything the admins may care to wire up through an Apache / NGINX proxy. And the basic job submission and status APIs have stablizied to the point that a client application should be able to consume nearly any version going forward.

Re: The art of high performance computing

#120

Post by johnnyanmac »

justin66 wrote: post #49
bluedino wrote: post #37

Earlier quoted context omitted.

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any k…

> The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Is it possible that pretty much any specialization, outside of the most common ones, engages in a lot of gatekeeping? I remember how difficult it appeared to be after I graduated to break into embedded systems (I never did). I persisted until I realized it doesn't even pay very well, comparatively.

Yes, but to varying degrees. I imagine the whole "fortune 500" deal probably gatekept more than it really needed to. While I don't think any specialization NEEDS a masters since 2-3 years of industry work in that field will do just as much 90% of the tie (a few may need PhD's, mostly for R&D labs), some can justify it more than others.

It's also cultural. From what I hear, the east cost US cares a lot more about prestige than the west coast that focuses a lot more on performance.

Post reply on HN