Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reservoir Sampling

samwho.dev

Post reply on HN
Print view View original

81–90 of 107 posts

Re: Reservoir Sampling

#81

Post by samwho »

eru wrote: post #56
samwho wrote: post #50

Earlier quoted context omitted.

I actually read that post on the alias method just the other day and was blown away. I think I’d like to try making a post on it. Wouldn’t be able to add anything that link hasn’t already said, but I think I can make it more accessible.

I have a few more topics we could cooperate on, if you are interested. https://claude.ai/public/artifacts/62d0d742-3316-421b-9a7b-d... has a 'very static' visualisation of sorting algorithms. Basically, we have a 2d plane, and we colour a pixel (x, y) black iff the sorting algorithm compares x with y when it runs. It's a resurrection (with AI) of an older project I was coding up manually at https://github.com/matthia…

I appreciate the offer (and your contributions in the comments here!) but collaborations are very difficult for me atm. Most of the work I do on these posts I do when I can steal time away from other aspects of my life, which can sometimes take weeks. I wouldn’t be a dependable collaboration partner.

Re: Reservoir Sampling

#82

Post by Matumio »

samwho wrote: post #3

Hello! o/ I’m the author of this post. Happy to answer any questions, and love to get feedback. The code for all of my posts can be found at https://github.com/samwho/visualisations and is MIT licensed, so you’re welcome to use it :)

The post looks interesting, but I'm still building my triangle castle so I haven't read it yet. This probably means I should read it later when I'm less distracted. Anyway, already love the layout and the playful visualisations.

Re: Reservoir Sampling

#83

Post by eru »

samwho wrote: post #81
eru wrote: post #56

Earlier quoted context omitted.

I have a few more topics we could cooperate on, if you are interested. https://claude.ai/public/artifacts/62d0d742-3316-421b-9a7b-d... has a 'very static' visualisation of sorting algorithms. Basically, we have a 2d plane, and we colour a pixel (x, y) black iff the sorting algorithm compares x with y when it runs. It's a resurrection (with AI) of an older project I was coding up manually at https://github.com/matthia…

I appreciate the offer (and your contributions in the comments here!) but collaborations are very difficult for me atm. Most of the work I do on these posts I do when I can steal time away from other aspects of my life, which can sometimes take weeks. I wouldn’t be a dependable collaboration partner.

No worries.

I'd mostly just appreciate a beta tester / beta reader.

Re: Reservoir Sampling

#84

Post by HelloNurse »

justanotheratom wrote: post #28

Great article and explanation. On a practical level though, this would be the last thing I would use for log collection. I understand that when there is a spike, something has to be dropped. What should this something be? I don't see the point of being "fair" about what is dropped. I would use fairness as a last resort, after trying other things: Drop lower priority logs: If your log messages have levels (debug, info…

You should drop or consolidate some entries if you can, but then the important entries that remain can still be too many and require random culling because anything is better than choking.

Fair reservoir sampling can be made unfair in controlled ways (e.g. by increasing the probability of retaining an entry if its content is particularly interesting); it competes with less principled biased random (or less than random) selection algorithms as a technique of last resort.

Re: Reservoir Sampling

#85

Post by samwho »

eru wrote: post #83
samwho wrote: post #81

Earlier quoted context omitted.

I appreciate the offer (and your contributions in the comments here!) but collaborations are very difficult for me atm. Most of the work I do on these posts I do when I can steal time away from other aspects of my life, which can sometimes take weeks. I wouldn’t be a dependable collaboration partner.

No worries. I'd mostly just appreciate a beta tester / beta reader.

Totally happy to do that! You’ll find where to contact me on my homepage. :)

Re: Reservoir Sampling

#86

Post by samwho »

DonHopkins wrote: post #75
samwho wrote: post #11

Earlier quoted context omitted.

Thank you so much! The dogs on the playing cards were commissioned just for this post. They’re all made by the wonderful https://www.andycarolan.com/ . The colour palette is the Wong palette that I learned about from https://davidmathlogic.com/colorblind/ . Oh, and you can pet the dogs. :)

So many nice touches that combine to be much more than the sum of the parts. Doe's bandana is cool, your dogs must worship you for your commitment to them! My only suggestion is a way to slow down or ^S the log to read the funny messages, since they were flying by so fast I could only get a glimpse, even with reservoir sampling. something something "needs more emojis"! ;)

Doe’s bandana is my attempt at tasteful solidarity and support. Glad you noticed it!

I did consider a pause button for the logs but it felt too unsubtle, and distracts from the content of the post. You could argue the log messages are already distracting, but I really wanted my own take on “reticulating splines.”

You can read how the messages are constructed here: https://github.com/samwho/visualisations/blob/main/reservoir...

Re: Reservoir Sampling

#87

Post by amw-zero »

codr7 wrote: post #58
magicalhippo wrote: post #9

We lived in a rural area when I was a kid. My dad told me once that his buddy had to measure the ptarmigan[1] population in the mountains each year as part of his job. He did this by hiking a fixed route, and at fixed intervals scare the birds so they would fly and count. The total count was submitted to some office which used it to estimate the population. One year he had to travel abroad when the counting had to be…

Never trust statistics. I once worked on a reservation system for some pretty big ski resorts. We were running late, working nights, and one of the last things we had to finish was the official statistics reports about number of guest nights etc that gets published by the government. Lets just say that the statistics that year had little to do with reality.

You’re confusing statistics with forecasting. We can and should trust statistics. We should just never trust their relation to future behavior.

Re: Reservoir Sampling

#89

Post by swiftcoder »

amw-zero wrote: post #87
codr7 wrote: post #58

Earlier quoted context omitted.

Never trust statistics. I once worked on a reservation system for some pretty big ski resorts. We were running late, working nights, and one of the last things we had to finish was the official statistics reports about number of guest nights etc that gets published by the government. Lets just say that the statistics that year had little to do with reality.

You’re confusing statistics with forecasting. We can and should trust statistics. We should just never trust their relation to future behavior.

Fairly sure the previous poster is describing statistics that were made up, rather than measured/sampled. Those are, for hopefully obvious reasons, not very trustworthy
Post reply on HN