Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sequential and parallel execution of long-running shell commands

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Sequential and parallel execution of long-running shell commands

#41

Post by Nukesor »

kjkjadksj wrote: post #24

What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.

You can probably achieve a good subset of its functionality in bash, it's just a nicer interface with a lot of configurability and several convenience features.

I'm generally a big fan of showing alternatives: https://github.com/Nukesor/pueue/?tab=readme-ov-file#similar...

Would you be willing to write a proper guide on how to do all of these things in bash? It would be great to have such a guide inside the Pueue wiki and link to it. It'll help people to make a more informed decision on whether they need this tool or not.

Re: Sequential and parallel execution of long-running shell commands

#42

Post by Nukesor »

veyh wrote: post #35

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally. [1] https://nats.io/

Interesting.

May I ask how many tasks you were managing with Pueue and what your usecase was?

I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier.

If there's a good usecase for it, I might consider switching to a more compact format.

Re: Sequential and parallel execution of long-running shell commands

#43

Post by Nukesor »

bionhoward wrote: post #10

This looks awesome and the README seems to undersell it. Would it be possible for you to put code examples all over the readme so we can see what you’re talking about without needing to dig into the codebase?

So, there's a wiki which explains many of the usecases.

I specifically didn't want to further bloat the README, as it's already super long as it is.

Re: Sequential and parallel execution of long-running shell commands

#44

Post by kjkjadksj »

seabrookmx wrote: post #29
kjkjadksj wrote: post #26

Earlier quoted context omitted.

Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.

Yes or I could use the bash `&&` but the issue is I need to know all the commands at the start. I want to be able to come back an hour later and easily add a command to the end of the chain.

You can do that by abrogating your pid

Re: Sequential and parallel execution of long-running shell commands

#45

Post by kjkjadksj »

Nukesor wrote: post #40
kjkjadksj wrote: post #26

Earlier quoted context omitted.

Could you just chain them in bash script? You can do it in a dumb way or you can even do it conditionally on the exit status of the previous rsync command.

Also, editing a command in the middle if you notice a mistake becomes tricky. Pueue also allows you to do stuff like dependencies, which get tricky in bash if a task depends on more than one tasks finishing.

If this ability is important to you then you can break your pipeline into individual files that aren’t going to be read until they are executed, giving you time to edit.

Re: Sequential and parallel execution of long-running shell commands

#46

Post by kjkjadksj »

pdimitar wrote: post #25
kjkjadksj wrote: post #21

Earlier quoted context omitted.

You can script up something similar to your multitail example using tmux.

Is it going to automatically close when all tasks complete?

That can be done with ending the command with something like “; exit 0”

Re: Sequential and parallel execution of long-running shell commands

#47

Post by martinthought »

"pueue follow " lets you see stdout or stderror of the specified task.

If one enqueues a single chain of tasks (no parallel tasks), is there a way to monitor stdout or stderror for the chain, without having to issue the follow command for each task at the time the task starts to run? This would provide better observability of what is running, as in a shell script with the tasks sequentially listed.

Re: Sequential and parallel execution of long-running shell commands

#48

Post by Nukesor »

kjkjadksj wrote: post #45
Nukesor wrote: post #40

Earlier quoted context omitted.

Also, editing a command in the middle if you notice a mistake becomes tricky. Pueue also allows you to do stuff like dependencies, which get tricky in bash if a task depends on more than one tasks finishing.

If this ability is important to you then you can break your pipeline into individual files that aren’t going to be read until they are executed, giving you time to edit.

Fair point, but to be honest, at this point it's just easier to do a:

pueue add 'rsync somestuff host:location'

And if I notice any problems, I just do a `pueue edit $id` and I'm good to go. It's just a lot more convenient than manually building pipelines with files that'll be executed.

It would be something different if this was about recurrent tasks that needed to be done, though. But for one-off stuff, your approach seems a bit cumbersome.

Re: Sequential and parallel execution of long-running shell commands

#49

Post by veyh »

Nukesor wrote: post #42
veyh wrote: post #35

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally. [1] https://nats.io/

Interesting. May I ask how many tasks you were managing with Pueue and what your usecase was? I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier. If there's a good usecase for it, I might consider switching to a more compact format.

I was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB.

zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though.

https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4...

Protobuf could be a pretty good alternative. It can be dumped to a human-readable format with the protoc cli.

Re: Sequential and parallel execution of long-running shell commands

#50

Post by dspillett »

Nukesor wrote: post #37
dspillett wrote: post #17

Earlier quoted context omitted.

> How is the pause and resume done? Perhaps by sending SIGSTOP and SIGCONT, much like hitting Ctrl+Z on the console and later running bg or fg . Note that this is not the same as Ctrl+S & Ctrl+Q on the console – that just pauses the output display not the process (though the process may subsequently pause if a buffer somewhere down the pipeline becomes full due to the terminal output pausing).

Yep, that's exactly how it's done :)

Thought it would be, but I didn't want to state that more authoritatively as I'd not bothered checking the docs/source. And I'm lazy like that.
Post reply on HN