I do all this with systemd user services.
Sequential and parallel execution of long-running shell commands
31–40 of 50 posts
Re: Sequential and parallel execution of long-running shell commands
#32What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.
Actually, the GNU xargs utility is the most efficient way to perform parallel execution on a large queue of input files.
GNU parallel is also an option, but it is written in Perl and has a larger footprint.
https://www.linuxjournal.com/content/parallel-shells-xargs-u...
Re: Sequential and parallel execution of long-running shell commands
#33Re: Sequential and parallel execution of long-running shell commands
#34Re: Sequential and parallel execution of long-running shell commands
#35Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.
[1] https://nats.io/
Re: Sequential and parallel execution of long-running shell commands
#36“Pueue is not designed to be a programmable (scriptable) task scheduler/executor.”
Any alternatives similar to Pueue with capabilities of above?
Re: Sequential and parallel execution of long-running shell commands
#37Under "Features": >Pause/resume tasks, when you need some processing power right NOW! How is the pause and resume done?
> How is the pause and resume done? Perhaps by sending SIGSTOP and SIGCONT, much like hitting Ctrl+Z on the console and later running bg or fg . Note that this is not the same as Ctrl+S & Ctrl+Q on the console – that just pauses the output display not the process (though the process may subsequently pause if a buffer somewhere down the pipeline becomes full due to the terminal output pausing).
Yep, that's exactly how it's done :)
Re: Sequential and parallel execution of long-running shell commands
#38“Pueue is not designed to be a programmable (scriptable) task scheduler/executor.” Any alternatives similar to Pueue with capabilities of above?
Not that I know of.
If anyone knows of stuff like that, please let me know, as this is regularly asked for :D.
I would be pretty stoked to be hired to build something like that, though.
Re: Sequential and parallel execution of long-running shell commands
#39Earlier quoted context omitted.
I was about to write that this has a different use case, because managing screen/tmux sessions is very manual. However the repo iself states > Pueue is not designed to be a programmable (scriptable) task scheduler/executor. The focus of pueue lies on human interaction. So I also don't really see its usecase and probably would opt for tmux instead. If you had many workers to run, but still do it manually, you would us…
>The focus of pueue lies on human interaction. How is it better than me having another window open and running my long running command there?
Well, if you're planning on running 10 commands sequentially that might take a few hours each, you would have to keep those windows open for quite a while.
See https://github.com/Nukesor/pueue/wiki/FAQ#what-can-i-use-it-...
Re: Sequential and parallel execution of long-running shell commands
#40I've wanted to write something like this for years! For me it's always long running rsync commands I want to chain. I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I'll be checking this out.
Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.
Also, editing a command in the middle if you notice a mistake becomes tricky.
Pueue also allows you to do stuff like dependencies, which get tricky in bash if a task depends on more than one tasks finishing.