Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Virtual SLURM HPC cluster in a Docker Compose

github.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#11

Post by throw0101c »

robot-wrangler wrote: post #3

Thanks for this! I went looking for something similar a while back and found nothing much. I'm guessing that the alternative to this tidy modern repository is a gigantic broken pile of ansible/chef/puppet that hasn't been touched in 10 years. Even surprisingly popular distributed-systems stuff is always really bad about "follow this 10 step copy/paste to deploy to EKS" but that's also obnoxious. In the first place, p…

> I'm guessing that the alternative to this tidy modern repository is a gigantic broken pile of ansible/chef/puppet that hasn't been touched in 10 years.

Not quite sure how well you looked, but there are a bunch of deployment systems for HPC, Ansible or otherwise:

* https://old.reddit.com/r/HPC/comments/1p4a3fq/what_imaging_s...

* My comment listing a bunch: https://news.ycombinator.com/item?id=46037792

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#12

Post by janeway »

Cool!

I have worked 100% in 3 comparable systems over the past 10 years. Can you access with ssh?

I find it super fluid to work on the HPC directly to develop methods for huge datasets by using vim to code and tmux for sessions. I focus on printing detailed log files constantly with lots of debugs and an automated monitoring script to print those logs in realtime; a mixture of .out .err and log.txt.

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#17

Post by IshKebab »

siliconpotato wrote: post #15
IshKebab wrote: post #13

I wish there was a sane modern alternative to SLURM. Futile hope though. My company is still using SGE.

Slurm is the modern alternative. We are using SGE too and slurm feels like the future.

Yeah unfortunately it still sucks. Actually to be fair it's probably fine for its intended use case: researchers interactively running one-off batch jobs on a university HPC cluster.

But I work in silicon and every company I've worked in uses SGE/SLURM for automated testing. SLURM absolutely sucks for that. They really want you to submit jobs as bash scripts, they can't handle a large number of jobs without using janky array jobs, submitting a job and waiting for it to finish is kind of janky. Getting the output anywhere except a file is difficult. Nesting jobs is super awkward and buggy. All the command line tools feel like they're from the 80s - by default the column widths are like 5 characters (not an exaggeration).

We even had an issue that SLURM uses 4 ports per job for the duration of the job, so you can't actually run more than a few thousand jobs simultaneously because the controller runs out of TCP ports!

I don't think it would actually be that hard to write a modern replacement. The difficult bit is dealing with cgroups. I won't hold my breath for anyone in the silicon industry to write it though. Hardware engineers can't write software for shit.

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#18

Post by siliconpotato »

IshKebab wrote: post #17
siliconpotato wrote: post #15

Earlier quoted context omitted.

Slurm is the modern alternative. We are using SGE too and slurm feels like the future.

Yeah unfortunately it still sucks. Actually to be fair it's probably fine for its intended use case: researchers interactively running one-off batch jobs on a university HPC cluster. But I work in silicon and every company I've worked in uses SGE/SLURM for automated testing. SLURM absolutely sucks for that. They really want you to submit jobs as bash scripts, they can't handle a large number of jobs without using jan…

> We even had an issue that SLURM uses 4 ports per job for the duration of the job, so you can't actually run more than a few thousand jobs simultaneously because the controller runs out of TCP ports!

That sounds concerning. Do you have a link to a bug report for this please? Is the tcp port problem on the compute node side or the controller side?

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#19

Post by IshKebab »

siliconpotato wrote: post #18
IshKebab wrote: post #17

Earlier quoted context omitted.

Yeah unfortunately it still sucks. Actually to be fair it's probably fine for its intended use case: researchers interactively running one-off batch jobs on a university HPC cluster. But I work in silicon and every company I've worked in uses SGE/SLURM for automated testing. SLURM absolutely sucks for that. They really want you to submit jobs as bash scripts, they can't handle a large number of jobs without using jan…

> We even had an issue that SLURM uses 4 ports per job for the duration of the job, so you can't actually run more than a few thousand jobs simultaneously because the controller runs out of TCP ports! That sounds concerning. Do you have a link to a bug report for this please? Is the tcp port problem on the compute node side or the controller side?

The controller side. I don't think it is a bug; that's just how they designed it.

They want you to use array jobs for large jobs, or submit jobs in a fire-and-forget way.

Re: Show HN: Virtual SLURM HPC cluster in a Docker Compose

#20

Post by ciclotrone »

janeway wrote: post #12

Cool! I have worked 100% in 3 comparable systems over the past 10 years. Can you access with ssh? I find it super fluid to work on the HPC directly to develop methods for huge datasets by using vim to code and tmux for sessions. I focus on printing detailed log files constantly with lots of debugs and an automated monitoring script to print those logs in realtime; a mixture of .out .err and log.txt.

You can access via SSH either with password or with keys.

Our reference cluster has long queuing times during busy hours and requires 2FA for access, so we had extra incentives to have a self-contained solution to run on our development machines.

Post reply on HN