At some point I made a tool called Orbit (here https://github.com/vertices-solutions/orbit) that works as a local interface to a Slurm.
It reached a stage when it does everything I need, but I don't know if it's a good fit to others. OP, if you want to run Slurm jobs with them feeling more local - come check it out!
Computational chemistry, Locally? My rant on SSH
21–30 of 45 posts
Re: Computational chemistry, Locally? My rant on SSH
#22OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily. Basically every issue they seem to have is solved by using VSCode + Remote development (Transfer…
The default VSCode configuration can cause problems in HPC clusters. VSCode likes to crawl the files it can see repeatedly. If you have a few users, it doesn't matter. But with many concurrent users and many files, it can destroy file system performance.
Re: Computational chemistry, Locally? My rant on SSH
#23> There is also a layer of friction between doing the computations and then analysing the logs that comp chemistry programs spit out. If you want to visualise orbitals or look at the outputs of your simulations, it's really awkward to run Python whilst being in the cluster, and the cluster has no GUI display. By the description it appears computational chemistry is stuck in batch processing era. They are yet to disco…
> it appears computational chemistry is stuck in batch processing era
I wouldn't call it "stuck," since it's one of the few fields where batch processing makes sense.
You generally run your jobs on a supercomputer that's being shared by a few hundred other people. You need a scheduling system to allocate resources (e.g., 5 jobs requesting 10 nodes for 8 hours, 1000 jobs requesting 0.5 node for 5 minutes, etc).
It's hard to tighten the workflow when the average calculation takes 2-3 hours of time.
Re: Computational chemistry, Locally? My rant on SSH
#24OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily. Basically every issue they seem to have is solved by using VSCode + Remote development (Transfer…
[deleted]
Re: Computational chemistry, Locally? My rant on SSH
#25> There is also a layer of friction between doing the computations and then analysing the logs that comp chemistry programs spit out. If you want to visualise orbitals or look at the outputs of your simulations, it's really awkward to run Python whilst being in the cluster, and the cluster has no GUI display. By the description it appears computational chemistry is stuck in batch processing era. They are yet to disco…
I’m a computational chemist who has been working on this kind of stuff for a while, and I will say it’s just a weird field. The overall issue is that it’s still very academic and very very fragmented. People have tried to make businesses out of this stuff but it has rarely worked, and there isn’t enough money/funding to sustain the level of work needed to really advance the field in that way. Especially now that gove…
> People have tried to make businesses out of this stuff but it has rarely worked
The only two startups I'm aware of who have managed to make compchem-as-a-service a viable business are Mat3ra and Materials Square. Both are a little rough in their own ways (to be expected from startups), but it's still interesting to see some of this move beyond university clusters.
Re: Computational chemistry, Locally? My rant on SSH
#26I'm so glad I left my previous job. As an SRE, I had to use about 10 tools regularly, and they all had 2 minute timeouts with "SSO" + 2fa. I literally spent about 30 minutes out of every hour just logging in to shit.
The security team wouldn't budge things up to 1 day even.
When you have 2 or 3 factors on every tool -- even things like prometheus. It gets excessive.
Re: Computational chemistry, Locally? My rant on SSH
#27OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily. Basically every issue they seem to have is solved by using VSCode + Remote development (Transfer…
>OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol
Sad that it's necessary, but it's still a shitty environment. That employers foist it on their employees doesn't make it better.
>I also wonder how they didn't come across X Forwarding at all?
Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
Re: Computational chemistry, Locally? My rant on SSH
#28OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily. Basically every issue they seem to have is solved by using VSCode + Remote development (Transfer…
The default VSCode configuration can cause problems in HPC clusters. VSCode likes to crawl the files it can see repeatedly. If you have a few users, it doesn't matter. But with many concurrent users and many files, it can destroy file system performance.
Every user should get their own vscode endpoint, on a severely underprovisioned VM with 10mb/sec access to the cluster filesystem.
Re: Computational chemistry, Locally? My rant on SSH
#29I'm so glad I left my previous job. As an SRE, I had to use about 10 tools regularly, and they all had 2 minute timeouts with "SSO" + 2fa. I literally spent about 30 minutes out of every hour just logging in to shit. The security team wouldn't budge things up to 1 day even. When you have 2 or 3 factors on every tool -- even things like prometheus. It gets excessive.
Security has no responsibility towards production, just for preventing breakins. Ergo, their process will always ratchet towards not letting anyone do anything.
Engine of a lawnmower, brakes of a Rolls-Royce, etc.
Re: Computational chemistry, Locally? My rant on SSH
#30OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily. Basically every issue they seem to have is solved by using VSCode + Remote development (Transfer…
>OP is not ready for corporate environments if "VPN + ssh + 2FA" already "feels like too much" lol Sad that it's necessary, but it's still a shitty environment. That employers foist it on their employees doesn't make it better. >I also wonder how they didn't come across X Forwarding at all? Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
> Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
During COVID I had the wonderful opportunity to remote into a work computer that had a Matlab license and watch the screen slowly paint in. Each keystroke took multiple seconds to display for me.