How to Shuffle and Sample on the Command-Line
blog.jpalardy.com
How to Shuffle and Sample on the Command-Line
1–10 of 21 posts
Re: How to Shuffle and Sample on the Command-Line
#2Nice! I've been using
sort -r | head -n100
Re: How to Shuffle and Sample on the Command-Line
#3Interesting. I did the same investigation myself a few years back, but was frustrated by the lack of the -r flag for shuf(1). It seems that's been added at some point recently (though many of my systems do not have it--GNU coreutils percolates slowly through older Debian/Ubuntu versions. :))
Good to know things are still getting better in coreutils!
Re: How to Shuffle and Sample on the Command-Line
#4 man shufRe: How to Shuffle and Sample on the Command-Line
#5Nice! I've been using sort -r | head -n100 but obviously this requires the entire file to be shuffled before printing the first 100 lines.
The -R option not being available on OS X, you might do something like
awk "BEGIN { srand($RANDOM) } { print int(rand() * 1000000), \$0 }" | sort -n | cut -d' ' -f2-
Re: How to Shuffle and Sample on the Command-Line
#6How does this handle files that are gigs in size? It looks to me like you load the entire file into memory and pass it along, shuffled.
Re: How to Shuffle and Sample on the Command-Line
#7How does this handle files that are gigs in size? It looks to me like you load the entire file into memory and pass it along, shuffled.
If you have an unknown input size and a finite number of requested output lines, you can still do it in O(n) and no additional memory, by having a steadily-decreasing chance of replacing one of the outputs with the next line.
Re: How to Shuffle and Sample on the Command-Line
#8Nice! I've been using sort -r | head -n100 but obviously this requires the entire file to be shuffled before printing the first 100 lines.
The -R option not being available on OS X, you might do something like awk "BEGIN { srand($RANDOM) } { print int(rand() * 1000000), \$0 }" | sort -n | cut -d' ' -f2- to shuffle an input
Re: How to Shuffle and Sample on the Command-Line
#9Earlier quoted context omitted.
The -R option not being available on OS X, you might do something like awk "BEGIN { srand($RANDOM) } { print int(rand() * 1000000), \$0 }" | sort -n | cut -d' ' -f2- to shuffle an input
http://bost.ocks.org/mike/shuffle/compare.html
Note that hnov's awk command is the equivalent of "sort (random order)" at that and shows good randomness properties in the plot. However, that link shows "sort (random comparator)" by default which looks terrible at randomly sorting lists. hnov's awk script should be suitable for most needs, though I'd tweak it a bit:
awk "{print rand(), $0}" | sort -g | cut -d' ' -f2-
Re: How to Shuffle and Sample on the Command-Line
#10man shuf
Yeah but what's sad is that
$ man -k shuffleI wish I knew about shuf. I've been using sort -R or using Python.