Pilosa: An open source, distributed index • Hacker News Board

Re: Pilosa: An open source, distributed index

#11

Post by fnordsensei » Fri, May 31, 2019, 12:59 PM UTC

dTal wrote: post #7

dmos62 wrote: post #5

Earlier quoted context omitted.

I found the use cases section the most informative. You can click on a use case to get a write-up. Here's a few excerpts from transportation: > Pilosa is a distributed bitmap index that sits on top of a data store. The key to understanding and then using Pilosa is converting data such that it is represented in ones and zeros. This dramatically reduces the size as well as accelerates query times. > For example, timest…

>converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.

Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas.

I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

Re: Pilosa: An open source, distributed index

#12

Post by ytklx » Fri, May 31, 2019, 1:02 PM UTC

dTal wrote: post #7

dmos62 wrote: post #5

Earlier quoted context omitted.

I found the use cases section the most informative. You can click on a use case to get a write-up. Here's a few excerpts from transportation: > Pilosa is a distributed bitmap index that sits on top of a data store. The key to understanding and then using Pilosa is converting data such that it is represented in ones and zeros. This dramatically reduces the size as well as accelerates query times. > For example, timest…

>converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.

Pilosa uses roaring bitmaps to store the data. Roaring bitmaps are compressed and performing bit operations on them is very efficient since they don't require uncompressing the whole bitmap to perform bit operations.

Re: Pilosa: An open source, distributed index

#13

Post by jaffee » Fri, May 31, 2019, 1:02 PM UTC

continuations wrote: post #6

How do you handle race conditions? E.g. an app updated the persistent store but crashed before it could update Pilosa?

Pilosa is best used in conjunction with something like Kafka with (e.g.) separate consumers for Pilosa and a persistent data store.

Re: Pilosa: An open source, distributed index

#14

Post by ytklx » Fri, May 31, 2019, 1:04 PM UTC

fnordsensei wrote: post #11

dTal wrote: post #7

Earlier quoted context omitted.

>converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.

Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

This is very similar to how Pilosa saves integers. https://www.pilosa.com/docs/latest/data-model/#bsi-range-enc...

Re: Pilosa: An open source, distributed index

#15

Post by maycotte » Fri, May 31, 2019, 1:05 PM UTC

sktrdie wrote: post #2

Not sure who this document is aimed to. It's not technical enough to appeal to programmers that are working closely with Pilosa. And it's not written in a way to make it easy to understand for people that don't know anything about Pilosa (such as myself). I mean a subtitle called "Time Quantum" is enough to make me confused. Would appreciate a more generic "what is this" intro if possible.

Great feedback. We could also do a better job of documenting the most frequent use cases which include high cardinality segmentation, personalization at scale, rapid data exploration/discovery, edge analytics, fast BI, threat detection, bioinformatics and more.

Re: Pilosa: An open source, distributed index

#16

Post by dTal » Fri, May 31, 2019, 1:07 PM UTC

fnordsensei wrote: post #11

dTal wrote: post #7

Earlier quoted context omitted.

>converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.

Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient, when an ordinary binary counter uses n = log2(amountOfBananasOwned).

Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.

Re: Pilosa: An open source, distributed index

#17

Post by fnordsensei » Fri, May 31, 2019, 1:12 PM UTC

dTal wrote: post #16

fnordsensei wrote: post #11

Earlier quoted context omitted.

Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient, when an ordinary binary counter uses n = log2(amountOfBananasOwned). Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.

Right! I meant in comparison to serializing data, which is what I (maybe incorrectly) assumed that the parent was referring to.

E.g., turning something like a JSON string of the same data into bits and sticking it in Pilosa.

Re: Pilosa: An open source, distributed index

#18

Post by joshuaellinger » Fri, May 31, 2019, 1:23 PM UTC

Technically, it is very interesting -- it uses Roaring Bitmaps under the hood and builds a query engine on it. So an easy way to think about it is that it maps categorical data into a giant compressible distributed bitmap.

I've been planning to see if I can (mis)use it as an OLAP replacement but I haven't had time to get to it.

Re: Pilosa: An open source, distributed index

#19

Post by ytklx » Fri, May 31, 2019, 1:27 PM UTC

dTal wrote: post #16

fnordsensei wrote: post #11

Earlier quoted context omitted.

Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient, when an ordinary binary counter uses n = log2(amountOfBananasOwned). Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.

Roaring bitmaps is quite good at compressing bits. Just as an example: Say, most monkeys have the same number of bananas. Pilosa doesn't store it as MonkeyCount bits, but just a few bytes.

Re: Pilosa: An open source, distributed index

#20

Post by jaffee » Fri, May 31, 2019, 1:30 PM UTC

joshuaellinger wrote: post #18

Technically, it is very interesting -- it uses Roaring Bitmaps under the hood and builds a query engine on it. So an easy way to think about it is that it maps categorical data into a giant compressible distributed bitmap. I've been planning to see if I can (mis)use it as an OLAP replacement but I haven't had time to get to it.

You definitely can... the feature set keeps growing. We have multi-field filtered GROUP BY now. It's amazing to see how flexible Roaring Bitmaps can be!