Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix command line queue utility

github.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Unix command line queue utility

#11

Post by shanemhansen »

cozzyd wrote: post #8
jessaustin wrote: post #7

Earlier quoted context omitted.

You don't need nq for that. Just kick off ./my_lame_analysis in the background whenever you want.

Yes but then they will execute at the same time instead of sequentially.

How about:

    ./my_awesome_analysis /data/complicated_file_with_long_name.dat &
    # do some stuf
    wait
    ./my_awesome_analysis /data/even_more_complicated_path.dat

Re: Unix command line queue utility

#12

Post by e40 »

shanemhansen wrote: post #11
cozzyd wrote: post #8

Earlier quoted context omitted.

Yes but then they will execute at the same time instead of sequentially.

How about: ./my_awesome_analysis /data/complicated_file_with_long_name.dat & # do some stuf wait ./my_awesome_analysis /data/even_more_complicated_path.dat

It's still not equivalent, and it's more cumbersome.

Re: Unix command line queue utility

#13

Post by kovek »

This stackoverflow answer[0] shows how you can queue jobs even after you start running other processes in your queue. Essentially, you can do:

  sleep 2; echo "A" && sleep 2 ; echo "B" && sleep 2 ; echo "C" 
  # Quickly Ctrl-Z
  fg && sleep 2 ; echo "D"
and you will see A\nB\nC\nD\n slowly printed out in your terminal.

[0]: http://superuser.com/a/345455

Re: Unix command line queue utility

#14

Post by justin_vanw » 

   $ mkfifo ~/.jobqueue
   $ parallel :::: ~/.jobqueue
now, from another terminal:

   echo ls > ~/.jobqueue
Tada!

for a remote job queue, use nc instead of a fifo

for a job queue with a buffer, use a file, and run $ parallel :::: Of course, this doesn't capture the context you are in, such as your current directory or the value of shell variables.

Re: Unix command line queue utility

#15

Post by stonogo »

icebraining wrote: post #10
dsl wrote: post #9

cat list.txt | xargs -n 1 -P 20 wget Queues up 20 concurrent downloads with wget, for example.

You could also use a for loop. But as others pointed out, that doesn't allow you to add items to the queue after it has started.

It does if you use a fifo instead of a text file. This is the difference between basic Unix literacy and actual Unix proficiency. I'm not being insulting; very few people take the time to learn the tools Unix provides. And naturally, a developer is going to be more interested in writing tools than most.

Re: Unix command line queue utility

#16

Post by giis »

justin_vanw wrote: post #14

$ mkfifo ~/.jobqueue $ parallel :::: ~/.jobqueue now, from another terminal: echo ls > ~/.jobqueue Tada! for a remote job queue, use nc instead of a fifo for a job queue with a buffer, use a file, and run $ parallel :::: Of course, this doesn't capture the context you are in, such as your current directory or the value of shell variables.

What's 'parallel'? Is it a *nix command? Looks like its not available on Fedora box

Re: Unix command line queue utility

#17

Post by teraflop »

giis wrote: post #16
justin_vanw wrote: post #14

$ mkfifo ~/.jobqueue $ parallel :::: ~/.jobqueue now, from another terminal: echo ls > ~/.jobqueue Tada! for a remote job queue, use nc instead of a fifo for a job queue with a buffer, use a file, and run $ parallel :::: Of course, this doesn't capture the context you are in, such as your current directory or the value of shell variables.

What's 'parallel'? Is it a *nix command? Looks like its not available on Fedora box

Yes, it's a GNU tool. Try "yum install parallel".

http://www.gnu.org/software/parallel/

Re: Unix command line queue utility

#18

Post by kovek »

justin_vanw wrote: post #14

$ mkfifo ~/.jobqueue $ parallel :::: ~/.jobqueue now, from another terminal: echo ls > ~/.jobqueue Tada! for a remote job queue, use nc instead of a fifo for a job queue with a buffer, use a file, and run $ parallel :::: Of course, this doesn't capture the context you are in, such as your current directory or the value of shell variables.

Here is an alternative:

  tty1$ mkfifo /tmp/jobqueue
  tty1$ eval $(tail -n 1 /tmp/jobqueue)
And then, in another terminal:

  tty2$ echo ls > /tmp/jobqueue

Re: Unix command line queue utility

#19

Post by icebraining »

stonogo wrote: post #15
icebraining wrote: post #10

Earlier quoted context omitted.

You could also use a for loop. But as others pointed out, that doesn't allow you to add items to the queue after it has started.

It does if you use a fifo instead of a text file. This is the difference between basic Unix literacy and actual Unix proficiency. I'm not being insulting; very few people take the time to learn the tools Unix provides. And naturally, a developer is going to be more interested in writing tools than most.

I don't think you can use a fifo with xargs, though it'd definitively work with a loop. Even with text files, using tail -f.

Re: Unix command line queue utility

#20

Post by iokanuon »

icebraining wrote: post #10
dsl wrote: post #9

cat list.txt | xargs -n 1 -P 20 wget Queues up 20 concurrent downloads with wget, for example.

You could also use a for loop. But as others pointed out, that doesn't allow you to add items to the queue after it has started.

    tail -n+0 -f list.txt | xargs -n 1 -P 20 wget
Post reply on HN