Live data from Hacker News

Algorithm No One Knows About (2016)

getkerf.wordpress.com

111–120 of 200 posts

Re: Algorithm No One Knows About (2016)

#111
post #84

So let me get this straight... This algorithm creates an initially empty array of size K with the DB internal ids of what would randomly be selected. Then it goes in order from 0 to the number of rows - 1, and for each one it goes e.g. "What are the chances that number 0 would get randomly picked out of N items if K items need to be picked?" Then, if the #0 passes the "random check", it gets added. Then it would go t…

No, what you described is O(n).

Re: Algorithm No One Knows About (2016)

#112

The last time I wanted an unpredictable, non-repeating sampling of 64-bit numbers, I used a symmetric cipher in CTR mode. Essentially, start with a random key and a random value, return the encrypted version of it, then the encrypted version of its increment, and so on. Returns every number in the range, with no predictability.

This works, but only if you're sampling the exact same space as the block size. For example, if your cipher outputs 64-bit blocks, but you want to sample in 2^32, you're back to the drawing board, because there's no way to truncate the cipher output without risking duplicates.

You can use the "hasty pudding trick" to truncate the cipher output without risking duplicates. It is inefficient if the block space of the cipher is much larger than the sampling space, though.

However, you can also use a Feistel network to create a block cipher for an arbitrary bit length. So, you can always bound the block space to be no more than twice the sampling space, which is okay.

Re: Algorithm No One Knows About (2016)

#113

The last time I wanted an unpredictable, non-repeating sampling of 64-bit numbers, I used a symmetric cipher in CTR mode. Essentially, start with a random key and a random value, return the encrypted version of it, then the encrypted version of its increment, and so on. Returns every number in the range, with no predictability.

The thing the Vitter algorithm does which this doesn't is that it produces the samples in order. This is why it's useful with tape drives, or sampling market data through the day: you can start at the beginning, and take the samples without ever having to seek backwards.

Re: Algorithm No One Knows About (2016)

#114

The last time I wanted an unpredictable, non-repeating sampling of 64-bit numbers, I used a symmetric cipher in CTR mode. Essentially, start with a random key and a random value, return the encrypted version of it, then the encrypted version of its increment, and so on. Returns every number in the range, with no predictability.

Choosing an initial random number, then keep modulo-incrementing it by a generator (a number larger than, and relatively prime to, the stack size) wouldn't work as well?

Re: Algorithm No One Knows About (2016)

#115
post #93
post #71

Earlier quoted context omitted.

In general, assuming that we have constant-time operations on arbitrarily large numbers (which we almost universally assume when doing these analyses, except in very specific fields), then hash structures do check membership and add items in O(1) time (since the time spent increasing the structure size is amortized fairly quickly). Additionally, the only problem with this approach is the extra O(k) space taken up by…

> then hash structures do check membership and add items in O(1) time (since the time spent increasing the structure size is amortized fairly quickly). Hashtables can never guarantee constant time operations. It is expected O(1) time, but it is completely possible that everything hashes to the same bucket and you get O(n) operations, no matter how much you rehash.

> Hashtables can never guarantee constant time operations.

A bit mask is a form of hash table and is guaranteed O(1) lookup for O(lg(Universe)) space.

When you have fixed universes, all kinds of options open up. I personally love vEB Trees, which can do set operations in O(lg(lg(U)))

Re: Algorithm No One Knows About (2016)

#116
post #13

While reading, I was thinking about LFSR [0] with suitable properties and period. Not strictly random, although LFSRs are commonly used for generating "random" numbers. [0]: https://en.wikipedia.org/wiki/Linear-feedback_shift_register

I was thinking the same. Depends on what "random" actually means. If you want to "pick one of these that you haven't already", a LFSR is great. Less so if you want truly random, or if anyone will start looking closely.

Re: Algorithm No One Knows About (2016)

#117
post #100
post #97

Some of the variable names used: i, j, t, qu1, S, n, N, U, X, y1, y2, V I know others have already commented on this but I think it's worth laying out all of the cryptic variables and asking: are some of these shorthand for well known math concepts that make it unnecessary to have a more descriptive name? I understand the use of temporary value holders like `i`, but have no clue what some of the others might be used…

I don't have an answer, but I remember Go programmers on Quora swearing up and down that (the Go convention of) using single-letter variables is ingenius and leads to great code ... although none of them could give a single example.

This is a bad take and odd place to criticize Go programmers.

Go convention is often short variable names, but the only place I know where single-letter variable names is encourage is for receiver methods. https://golang.org/doc/effective_go.html#methods

In that case the definition of the single-letter variable is literally a line or two of code away.

Re: Algorithm No One Knows About (2016)

#118
post #100

Earlier quoted context omitted.

I don't have an answer, but I remember Go programmers on Quora swearing up and down that (the Go convention of) using single-letter variables is ingenius and leads to great code ... although none of them could give a single example.

This is a bad take and odd place to criticize Go programmers. Go convention is often short variable names, but the only place I know where single-letter variable names is encourage is for receiver methods. https://golang.org/doc/effective_go.html#methods In that case the definition of the single-letter variable is literally a line or two of code away.

So is HN a good place to ask, and if so, are you able to find a perspicuous example highlighting the superior code quality?

Re: Algorithm No One Knows About (2016)

#119
post #21

Earlier quoted context omitted.

Trivial algorithms (like one implemented in Python) slow down when k is close to n. Or when n is large, then it is not possible to use bitmap to mark dealt cards.

What's preventing someone from implementing a non-trivial algorithm in Python?

That was not at all what I meant.

Re: Algorithm No One Knows About (2016)

#120

The last time I wanted an unpredictable, non-repeating sampling of 64-bit numbers, I used a symmetric cipher in CTR mode. Essentially, start with a random key and a random value, return the encrypted version of it, then the encrypted version of its increment, and so on. Returns every number in the range, with no predictability.

Why not simply increment the CTR IV with a monotonically increasing counter?
Post reply on HN