Live data from Hacker News

Computer scientists invent an efficient new way to count

quantamagazine.org

181–190 of 299 posts

Re: Computer scientists invent an efficient new way to count

#181
IDK, if my explanation is correct, but I do believe it is. I t goes as follow.

Imagine that you have a container of potential limitless capacity. The container starts with smalls capacity, equal to the real limited capacity that the real algorithm uses. As you add elements, when the container is full, its capacity is doubled, but all elements are then placed in a random position.

When you're done, you're told the occupancy of the subset of the large container corresponding to the initial size and how many times the container size was doubled. Multiplying that occupancy by the power of two of the number of doubling gives you an approximation of the real size.

The small catch is that in the actual algorithm, due to the discarding, the final number of elements, the occupancy, is somewhat erroneous.

EDIT

Another way to say this: you got a container of limited capacity S. When full, you "virtually" double its size and then randomly move elements over the full "imaginary" size of the virtual container. So after the first filling, you end up with about 1/2 the elements. After the second filling 1/4, etc. Also, since now your "virtual" container is larger, when you add a new element, there is only 1/2^n the it will be place inside your limited-capacity view of the entire virtual container.

At the end, the approximate real count is the number of elements you got multiplied by 2 to the power of the number of size doubling.

Again, it is as if you have a small window into a limitless container.

Re: Computer scientists invent an efficient new way to count

#182
post #176

The algorithm uses less memory, but more CPU time because of rather frequent deletions, so it's a tradeoff, not just generally better algorithm, as article may suggest.

the list is small so the cost of deletions should be small.

Re: Computer scientists invent an efficient new way to count

#183

Earlier quoted context omitted.

In python, you can simply substitute `A` with an iterable or generator object, which can be a of unknown length.

But for this algorithm, you need to know the total length ("m") to set the threshold for the register purges. Does it still work if you update m as you go?

You actually don't need to do that part in the algorithm. If you don't know the length of the list, you can just choose a threshold that seems reasonable and calculate the margin of error after you're done processing. (or i guess at whatever checkpoints you want if it's continuous)

In this example, they have the length of the list and choose the threshold to give them a desired margin of error.

Re: Computer scientists invent an efficient new way to count

#184
post #178

I was involved with implementing the DNF volume counting version of this with the authors. You can see my blog post of it here: https://www.msoos.org/2023/09/pepin-our-probabilistic-approx... And the code here: https://github.com/meelgroup/pepin Often, 30% of the time is spent in IO of reading the file, that's how incredibly fast this algorithm is. Crazy stuff. BTW, Knuth contributed to the algo, Knuths' notes: https…

That’s really interesting and thanks for sharing.

I am very curious about the extraordinarily gifted. What made you think Knuth is crazy good? Was there a particularly moment? Was it how fast he groked ideas? Was it his ability to ELI5?

Re: Computer scientists invent an efficient new way to count

#185

There is a big practicality problem I see with this algorithm. The thresh defined in the paper relies on the length of the stream. It seems to me that in a scenario where you have a big enough data set to desire a solution that doesn't just store every unique value you don't know the length. I did not make it through all the proofs but I believe they use the fact that the defined threshold has the length in it to pro…

If you don't know the length of the stream in advance, you can just calculate the margin of error when you're done, no?

Re: Computer scientists invent an efficient new way to count

#186

Earlier quoted context omitted.

> the terms are used interchangeably in most contexts Counting and estimating are not used interchangeably in most contexts. > because in the real world all counting methods have some nonzero error rate. The possibility that the counting process may be defective does not make it an estimation. > We talk about ‘counting votes’ in elections, for example, yet when things are close we perform ‘recounts’ which we fully ex…

So, IEEE floating point doesn’t support ‘addition’ then.

IEEE 754 defines an exact binary result for the addition of any two floats.

That this bit-identical result is not the same operation as addition of real numbers is irrelevant, because floats aren't reals.

f1 + f2 is not an estimation. Even treating it as an approximation will get you into trouble. It's not that either, it's a floating-point result, and algorithms making heavy use of floating point had better understand precisely what f1 + f2 is going to give you if they want to obtain maximum precision and accuracy.

Re: Computer scientists invent an efficient new way to count

#187

Earlier quoted context omitted.

In python, you can simply substitute `A` with an iterable or generator object, which can be a of unknown length.

But for this algorithm, you need to know the total length ("m") to set the threshold for the register purges. Does it still work if you update m as you go?

You would want to calculate the threshold by choosing your target epsilon and delta and an 'm' equal to the largest conceivable size of the stream. Fortunately, the threshold increases with log(m), so it's inexpensive to anticipate several orders of magnitude more data than necessary. If you wanted, you could work backwards to calculate the actual 'epsilon' and 'delta' values for the actual 'm' of the stream after the fact.

Re: Computer scientists invent an efficient new way to count

#188
post #11

Earlier quoted context omitted.

tbh, the title (and introduction) did a lot to dissuade me from finishing the (really good) article. It was actually informative, why dress it as a SEO blogspam?

Presumably so it is optimised for search engines and people find it. Publishers generally have good data about where their audience comes from. They wouldn't do this if it wasn't the best way they know of to maximise readership.

I've been following Quanta for some time, I'm sure they don't care about SEO and number of visitors, they care about the quality of texts. They write for the general audience, and even though they try to preserve the scientific accuracy, sometimes their explanations may seem oversimplified and even a bit confusing when you come from the same field. It's not clickbait, it's their popular science style.

Re: Computer scientists invent an efficient new way to count

#189
post #146

Earlier quoted context omitted.

I agree with you, but we are talking theory here. The algorithm doesn't count, it estimates. You can make an algorithm that counts, you can make an algorithm that estimates, this is the second.

Estimation is counting with error bars. Frankly, most of what you consider counting in your comment needs error bars - ask anyone who operated an all-cash cash-register how frequently end-of-day reconciliation didn't match the actual cash in the drawer (to the nearest dollar.) The following is a list from my personal experience - of presumably precisely countable things that didn't turn out to be the case: the number…

Counting is a subset of estimation, not a synonym.

If I estimated the number of quarters in a stack by weighing them, that would be different from estimating the number of quaters in a stack by counting them. Both methods of estimation have error bars.

The list you provide is of categories that don't have clear definitions. If you have a sufficiently clear definition for a category given your population, it has a precise count (though your counting methodologies will still be estimates.) If your definition is too fuzzy, then you don't actually have a countable set.

Re: Computer scientists invent an efficient new way to count

#190

There is a big practicality problem I see with this algorithm. The thresh defined in the paper relies on the length of the stream. It seems to me that in a scenario where you have a big enough data set to desire a solution that doesn't just store every unique value you don't know the length. I did not make it through all the proofs but I believe they use the fact that the defined threshold has the length in it to pro…

If you don't know the length of the stream in advance, you can just calculate the margin of error when you're done, no?

You just solve for epsilon. That's what I did: https://news.ycombinator.com/user?id=cb321
Post reply on HN