Dimsum is like head or tail, but pulls randomly from the entire file or stream
1–10 of 31 posts
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#2uses reservoir sampling -http://en.wikipedia.org/wiki/Reservoir_sampling
(so it presumably consumes the entire stream before giving any results; any alternative i can think of would not be "really random" unless you knew the length of the stream in advance).
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#3uses reservoir sampling - http://en.wikipedia.org/wiki/Reservoir_sampling (so it presumably consumes the entire stream before giving any results; any alternative i can think of would not be "really random" unless you knew the length of the stream in advance).
Yep, though a feature request I've put in is to respond to a ctl-c by producing the results from the stream so far... that way if it's taking a while on a large file you can interrupt and still get something useful.
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#4uses reservoir sampling - http://en.wikipedia.org/wiki/Reservoir_sampling (so it presumably consumes the entire stream before giving any results; any alternative i can think of would not be "really random" unless you knew the length of the stream in advance).
the magic of reservoir sampling is the memory footprint is the size of the output while being completely random. In this particular implementation the order of the output is slightly biased but each row has an equal chance of being in the output.
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#5uses reservoir sampling - http://en.wikipedia.org/wiki/Reservoir_sampling (so it presumably consumes the entire stream before giving any results; any alternative i can think of would not be "really random" unless you knew the length of the stream in advance).
the magic of reservoir sampling is the memory footprint is the size of the output while being completely random. In this particular implementation the order of the output is slightly biased but each row has an equal chance of being in the output.
but yes. it has to wait until the stream is done before producing any output.
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#6uhhh. You mean like shuf -n NNNN ?
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#7uhhh. You mean like shuf -n NNNN ?
I wonder if the implementation of shuf would handle very large input efficiently? Reservoir sampling wouldn't need to keep the whole input in memory, which could be an advantage. But I don't know how shuf works.
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#8uhhh. You mean like shuf -n NNNN ?
or "sort -R"...
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#9uses reservoir sampling - http://en.wikipedia.org/wiki/Reservoir_sampling (so it presumably consumes the entire stream before giving any results; any alternative i can think of would not be "really random" unless you knew the length of the stream in advance).
Yep, though a feature request I've put in is to respond to a ctl-c by producing the results from the stream so far... that way if it's taking a while on a large file you can interrupt and still get something useful.
[deleted]
Re: Dimsum is like head or tail, but pulls randomly from the entire file or stream
#10uhhh. You mean like shuf -n NNNN ?
I wonder if the implementation of shuf would handle very large input efficiently? Reservoir sampling wouldn't need to keep the whole input in memory, which could be an advantage. But I don't know how shuf works.
Doesn't look like it. I just tried running "yes | shuf -n 1" (using the latest version of GNU coreutils, 8.20) and its memory consumption increased steadily until I killed it.
It seems like this would be a really useful improvement, and I'm surprised that it doesn't already seem to have been requested on the coreutils issue tracker.