How does this handle files that are gigs in size? It looks to me like you load the entire file into memory and pass it along, shuffled.
See: reservoir sampling [1]
How to Shuffle and Sample on the Command-Line
11–20 of 21 posts
Re: How to Shuffle and Sample on the Command-Line
#12Interesting. I did the same investigation myself a few years back, but was frustrated by the lack of the -r flag for shuf(1). It seems that's been added at some point recently (though many of my systems do not have it--GNU coreutils percolates slowly through older Debian/Ubuntu versions. :)) Good to know things are still getting better in coreutils!
Oh, nice, on Fedora 23 beta, I can simulate die rolls
shuf -r -z -n 100 -e 1 2 3 4 5 6;echo -e "\n"
shuf -r -z -n 100 -e 3 3 3 3 3 6;echo -e "\n"
Re: How to Shuffle and Sample on the Command-Line
#13man shuf
It's not the documentation, it's knowing which tools exist in the first place.
Re: How to Shuffle and Sample on the Command-Line
#14Nice! I've been using sort -r | head -n100 but obviously this requires the entire file to be shuffled before printing the first 100 lines.
-r gives a random hash. So it will do the wrong thing in the face of repeated lines. (Either you get all instances of a repeated line or none.)
Re: How to Shuffle and Sample on the Command-Line
#15Interesting. I did the same investigation myself a few years back, but was frustrated by the lack of the -r flag for shuf(1). It seems that's been added at some point recently (though many of my systems do not have it--GNU coreutils percolates slowly through older Debian/Ubuntu versions. :)) Good to know things are still getting better in coreutils!
Oh, nice, on Fedora 23 beta, I can simulate die rolls shuf -r -z -n 100 -e 1 2 3 4 5 6;echo -e "\n" And rolls of a non-transitive (Grime) die shuf -r -z -n 100 -e 3 3 3 3 3 6;echo -e "\n" Which is timely. I sometimes forget how flexible the terminal prompt is...
It's pseudo-random though, right?
Re: How to Shuffle and Sample on the Command-Line
#16On Windows, Get-Random handles the basic cases nicely
> 1..100 | Get-Random
72
> 1..100 | Get-Random -count 3
16
96
56
Re: How to Shuffle and Sample on the Command-Line
#17Earlier quoted context omitted.
Oh, nice, on Fedora 23 beta, I can simulate die rolls shuf -r -z -n 100 -e 1 2 3 4 5 6;echo -e "\n" And rolls of a non-transitive (Grime) die shuf -r -z -n 100 -e 3 3 3 3 3 6;echo -e "\n" Which is timely. I sometimes forget how flexible the terminal prompt is...
It's pseudo-random though, right?
Oh, I'd imagine so. Good enough for illustrative purposes and for catching any gross errors in my arithmetic when analysing the games. Not good enough for anything 'real'.
Re: How to Shuffle and Sample on the Command-Line
#18I recently discovered shuf since I needed to shuffle a fairly large number of URL's in a file to allow multiple processes to work through them in parallel (yes I could have actually done this using a queue and a producer/consumer, but this was a one time deal so it was faster to just throw a bit more hardware at it). What I was amazed by is that it took 'cat urls.txt | shuf > new-urls.txt' just about a second to complete even thoug the original file was about 1GB. How does it work so incredibly fast?
Re: How to Shuffle and Sample on the Command-Line
#19Re: How to Shuffle and Sample on the Command-Line
#20man shuf
oh sorry, I mean
http://linux.die.net/man/1/shuf