Live data from Hacker News

Sequential and parallel execution of long-running shell commands

github.com

21–30 of 50 posts

Re: Sequential and parallel execution of long-running shell commands

#21
post #7

I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others. Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be execut…

You can script up something similar to your multitail example using tmux.

Re: Sequential and parallel execution of long-running shell commands

#22
post #12

I usually just need to run one big interactive job on a server, in which case I just do tmux my-command Ctrl-b d (detach) logout, go to bed, sleep, whatever log back into the server, tmux attach Would there be an advantage to using this over that?

Probably not in practice. You do save that ctrl-b keypress tho.

Ampersand also works, no tmux needed

Re: Sequential and parallel execution of long-running shell commands

#25
post #7

I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others. Seems like it has everything I need, the most crucial ones being separate queues / worker pools (f.ex. I only want the tasks that might involve source compilation to be execut…

You can script up something similar to your multitail example using tmux.

Is it going to automatically close when all tasks complete?

Re: Sequential and parallel execution of long-running shell commands

#26

I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain. I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.

Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.

Re: Sequential and parallel execution of long-running shell commands

#27
post #6

Earlier quoted context omitted.

I was about to write that this has a different use case, because managing screen/tmux sessions is very manual. However the repo iself states > Pueue is not designed to be a programmable (scriptable) task scheduler/executor. The focus of pueue lies on human interaction. So I also don't really see its usecase and probably would opt for tmux instead. If you had many workers to run, but still do it manually, you would us…

The concurrency controls are not present using the tmux method. Though if I am running enough tasks for this to be an issue then something odd is happening. The queue being persisted and surviving crashes could be useful, and isn't the case with tmux unless you manually script it up. Though it could be painful if the crash leaves things remaining tasks depend upon in an odd state… The task tree could be useful: sched…

You can do this in pure bash I believe. E.g. when you launch a command you can capture its pid then wait for that to exit as you expect before running further jobs.

Re: Sequential and parallel execution of long-running shell commands

#28
post #12

I usually just need to run one big interactive job on a server, in which case I just do tmux my-command Ctrl-b d (detach) logout, go to bed, sleep, whatever log back into the server, tmux attach Would there be an advantage to using this over that?

Your usecase reminds me of GNU screen

They're both terminal multiplexers. In my experience tmux is a much better experience though

Re: Sequential and parallel execution of long-running shell commands

#29

I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain. I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.

Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.

Yes or I could use the bash `&&` but the issue is I need to know all the commands at the start. I want to be able to come back an hour later and easily add a command to the end of the chain.

Re: Sequential and parallel execution of long-running shell commands

#30
A similar tool I highly recommend: https://github.com/justanhduc/task-spooler

At first I thought it would just be a one-off tool I used for one of my projects, not until I discovered later that it has everything I need and became my daily driver ever since.

Post reply on HN