Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design of GNU Parallel (2015)

gnu.org

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Design of GNU Parallel (2015)

#61

Post by mianos »

I once replaced a 10 machine Hadoop cluster job with a python script and parallel on my laptop because I didn't want to wait for hours for it to finish.

The i7 on my laptop with quite a few CPUS/threads and a few optimisations got the job finished in 10 minutes.

(I later put the Hadoop use on my resume, not the GNU parallel. That's the joke of modern job hunting. There is no interested in what you did, just buzzwords and leetcode. Luckily there are still a few places that value real work or I'd be too old to get a job. :) )

Re: Design of GNU Parallel (2015)

#62

Post by ajsnigrutin »

fmajid wrote: post #31
ajsnigrutin wrote: post #24

Earlier quoted context omitted.

Yep, that's the great thing about perl... take a 20 year old script and it still works today. In comparison, if they used python, they'd be using python 2.2.

That's basically a side-effect of Perl being a dead language, frozen because Perl 6 will never happen. It's surprisingly hard to eradicate, however.

Not making breaking changes every few years doesn't mean that the language is dead. It's still being developed and new versions of perl are still coming out.

Re: Design of GNU Parallel (2015)

#63

Post by rockwotj »

codetrotter wrote: post #40
rockwotj wrote: post #25

I recently used parallel to write a 1TB data file for testing using all cores seq 0 10000 | parallel dd if=/dev/urandom of=/mnt/foo/input bs=10M count=10 seek={}0

Was it noticeably different from dd if=/dev/urandom of=/mnt/foo/input bs=10M count=100000 in the amount of time that it took?

Yes, I had 16 cores and I gave up on the this version after several minutes. I don't remember the disk throughput difference but it was significant

Re: Design of GNU Parallel (2015)

#64

Post by BooneJS »

Alifatisk wrote: post #46
BooneJS wrote: post #30

Before GNU Parallel I used to use Ruby's workers and job queue to keep ${N} cores busy with work. It sorta worked like GNU parallel but was quite basic. I've since switched to using GNU Parallel. Stable code I don't have to write doesn't have to be maintained... not to mention it has more features than I normally supported.

What did you use exactly? I am curious, Resque? Sidekick?

Ruby's Queue structure to push work, and Thread for spinning up workers based on the number of cores on the machine. Main thread would push all commands to run to the Queue, followed by ${N} shutdown hints, and ${N} Threads would pick them off in a while loop that would only stop when it saw a shutdown command. Once the last thread consumed the last shutdown hint, all threads were done and the script would exit. This was barely one step beyond a bash script that backgrounded all tasks at once and swamped a host until it slowly finished up.

Re: Design of GNU Parallel (2015)

#65

Post by xyzzy_plugh »

RhysU wrote: post #58
xyzzy_plugh wrote: post #53

Earlier quoted context omitted.

It's nonsense because a utility like parallel shouldn't require state, let alone state used only to disable a nag message. It's far less annoying to simply patch out the nag. As others point out, it's further annoying because it doesn't even make any sense to begin with. If it was asking for donations or something I could maybe even get behind it, but the current message is pretentious and useless. It serves no real…

Then hop on the mailing list and suggest he set up a donation drop and donate.

Arguing with Ole is a waste of time. My parallel is patched, I don't care any longer.

Re: Design of GNU Parallel (2015)

#66

Post by chubot »

ajsnigrutin wrote: post #24
ketzu wrote: post #16

This was quite interesting to look through! Perl 5.8.0 is over 20 years old ( https://dev.perl.org/perl5/news/2002/07/18/580ann/ ) while centOS 3.9 was released in 2007! At the same time it seems not-that-old and ancient. My personal anecdote with gnu parallel was running into it while working in academia. It worked well and saved me some time, but I felt that it was unreasonable of a tool to ask for a citation to pa…

Yep, that's the great thing about perl... take a 20 year old script and it still works today. In comparison, if they used python, they'd be using python 2.2.

Python 2.7 was released in 2010, and is even more frozen than Perl!

It still works, though you would have to archive/vendor dependencies

Re: Design of GNU Parallel (2015)

#67

Post by chubot »

remram wrote: post #29
mhh__ wrote: post #23

Earlier quoted context omitted.

assembly optimizations for starting processes?

Maybe for reading the input, splitting it, and assembling the possibly-very-long argument lists passed to the processes.

Those things are all very fast compared to starting a process

Re: Design of GNU Parallel (2015)

#68

Post by remram »

chubot wrote: post #67
remram wrote: post #29

Earlier quoted context omitted.

Maybe for reading the input, splitting it, and assembling the possibly-very-long argument lists passed to the processes.

Those things are all very fast compared to starting a process

Command lines can be very long, so you can potentially read a million lines between executing processes.

Re: Design of GNU Parallel (2015)

#69

Post by fbdab103 »

KronisLV wrote: post #50
fbdab103 wrote: post #44

Earlier quoted context omitted.

This is pretty crafty. I do not know supervisor well enough - if one of the services fail, can you engineer supervisor to also crash so that it would bubble up to the container infrastructure? My understanding is that standard supervisor would let the process die and/or restart the service.

Supervisor allows you to have event listeners (e.g. for processes quitting/crashing), so you can use those to achieve that and kill supervisor itself. Here's an example of people doing just that: https://gist.github.com/tomazzaman/63265dfab3a9a61781993212f...

Neato. Do not have an immediate use case for this, but definitely something I will consider for the future.

Re: Design of GNU Parallel (2015)

#70

Post by adastra22 »

rockwotj wrote: post #63
codetrotter wrote: post #40

Earlier quoted context omitted.

Was it noticeably different from dd if=/dev/urandom of=/mnt/foo/input bs=10M count=100000 in the amount of time that it took?

Yes, I had 16 cores and I gave up on the this version after several minutes. I don't remember the disk throughput difference but it was significant

This should be I/o limited though.
Post reply on HN