Making Postgres Bloom

Post by anarazel » Fri, Jul 31, 2015, 2:48 AM UTC

striking wrote: post #16

usman-m wrote: post #15

Earlier quoted context omitted.

It probably uses a HyperLogLog--the 2% error rate kind of gives it away. Bloom filters approximate set membership queries, HyperLogLogs approximate set cardinality queries. COUNT DISTINCT is a set cardinality query. We actually support a HyperLogLog backed COUNT DISTINCT aggregate too: http://docs.pipelinedb.com/aggregates.html#general-aggregate...

Consider my metaphorical hat eaten. Thanks for the cool tools! I'm currently working with Postgres and this looks like a great thing to add to the mix.

Post by [deleted] » Fri, Jul 31, 2015, 4:28 AM UTC

danneu wrote: post #11

pbnjay wrote: post #5

Don't get me wrong, I love Postgres and use it in pretty much all of my projects... but for this kind of application it's not very well suited. Leave your relational data for the database and use something more efficient! Redis comes with both bitfields (see http://redis.io/commands/bitcount ) and hyperloglog counters (see http://redis.io/commands/pfcount ), requires almost no setup and has very minimal overhead.

Classic premature optimization. "Just add another database!"

Post by teddyh » Fri, Jul 31, 2015, 11:40 AM UTC

gleb wrote: post #8

The example doesn't quite make sense: SELECT user_id IN (SELECT DISTINCT user_id FROM user_actions); is not valid SQL. You may mean something like: SELECT 123 IN (SELECT DISTINCT user_id FROM user_actions); which is a strange query, as it's equivalent to: SELECT 123 IN (SELECT user_id FROM user_actions);

  SELECT EXISTS (SELECT 1 FROM user_actions WHERE user_id = 123);