Live data from Hacker News

Scalable Bloom Filters (2007) [pdf]

citeseerx.ist.psu.edu

11–20 of 39 posts

Re: Scalable Bloom Filters (2007) [pdf]

#11
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

Any in fact anything that sounds sort of computer sciencey.

Re: Scalable Bloom Filters (2007) [pdf]

#12
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

I thought I might have a use for them in practice, but it turns out they are only useful for sparse datasets — if your dataset can be fully populated, a bitmap is more efficient (for any reasonable degree of accuracy).

Re: Scalable Bloom Filters (2007) [pdf]

#13
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

Bloom filters are popular mostly because they’re clever, but they can also be useful.

E.g. Google Chrome’s safe browsing feature used Bloom filters to check whether a site might be flagged for malware, and if necessary query Google to check if it is actually flagged. It would have been impractical to download the full safe browsing dataset to every client, and also impractical to query Google for every single site visited. Bloom filters were a good solution (note: they’ve since moved to a custom data structure[1] which is also interesting.)

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=71832

Re: Scalable Bloom Filters (2007) [pdf]

#14
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

Extremely useful for anyone doing real-time analytics or filtering. Answering "has this person visited this page before" or "how many distinct people have visited this page/website" can be done exactly with map-reduce or streaming pipelines, but if you want an immediate answer with low latency and can tolerate false negatives/bounded inaccuracy, you can simply keep a Bloom filter or HyperLogLog in memory. For example, see the top comment here: https://news.ycombinator.com/item?id=12124722

Re: Scalable Bloom Filters (2007) [pdf]

#16
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

> Personally I've never found a use for either of them in practice.

One of my servers batch downloads approximately 550GB of data per day. I use a bloom filter to deduplicate the file significantly faster than naive Unix sort or awk ‘!X[$0]++’. It also has superior asymptotic complexity.

Re: Scalable Bloom Filters (2007) [pdf]

#17
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

They have one use: as a fast check for whether you should bother with a slower check. Bloom filters are basically a sort of a pre-cache.

Re: Scalable Bloom Filters (2007) [pdf]

#18
post #2

Hacker news loves upvoting articles about bloom filters (and Bayesian probability - its on the front page again this evening.) Personally I've never found a use for either of them in practice.

They have one use: as a fast check for whether you should bother with a slower check. Bloom filters are basically a sort of a pre-cache.

They actually have other uses, like measuring file similarity in sdhash and dimensionality reduction in machine learning.

Re: Scalable Bloom Filters (2007) [pdf]

#19
this is a kind of tangential comment/rant :

but to me it seems that research papers _must_ be, for lack of a better term, runnnable. i would, and hopefully others as well, like to, replicate all these wonderful results that are advertised in these papers. without that, they are all just advertisements of scholarship rather than scholarship themselves. a set of instructions + environment which generated these figures would be very welcome.

on the subject of bloom filters, have a look at this one: https://www3.cs.stonybrook.edu/~ppandey/files/p775-pandey.pd... (A General-Purpose Counting Filter: Making Every Bit Count)

Re: Scalable Bloom Filters (2007) [pdf]

#20
post #9

This should also be marked with a year. A cursory google search has StackOverflow answers from 2013. A lack of a year label implicitly suggests that it’s new, e.g. Hacker News. Please add a tag with the appropriate year.

Looks like it's from 2007: https://www.sciencedirect.com/science/article/pii/S002001900...

OK, added. Thanks!
Post reply on HN