FeatureBase: Open-Source, Real-Time Database Built on Roaring Bitmaps • Hacker News Board

Re: FeatureBase: Open-Source, Real-Time Database Built on Roaring Bitmaps

#11

Post by kordlessagain » Wed, Sep 14, 2022, 8:49 PM UTC

JAA1337 wrote: post #8

kurosknight wrote: post #7

Earlier quoted context omitted.

They are useful for categorical variables. For example, is a record in the "Likes motorcycles" category? They are fast because (well, one reason) bitwise logical operations are very fast for CPUs to do. Adtech is an example of a sector that benefits from this...they slice and dice datasets a lot to target ad campaigns and such. Being able to do that quickly is useful.

So are you saying that the data is stored in categories which allows for those types of lookups to run faster? Do you have specifics on how the design of a bitmap based database achieves this? How does it maintain these relationships? Just through 0 and 1's? I guess it's easy for me to visualize both row and column based. Im struggling with the bitmaps concept.

Here's a good write up on some of what you are asking in their blog: https://www.featurebase.com/blog/bitmaps-making-real-time-an...

Re: FeatureBase: Open-Source, Real-Time Database Built on Roaring Bitmaps

#12

Post by kordlessagain » Wed, Sep 14, 2022, 8:56 PM UTC

elina123 wrote: post #2

any ideas on real life Use Cases?

From what I understand, machine learning models may use this in ETL pipelines as well as serving as part of the models themselves. There's an article on that here: https://medium.com/analytics-and-data/overview-of-the-differ...

FeatureBase could be the "feature store" in the middle of the batch prediction section's diagram, or simply be a drop-in replacement for the model's registry.

Re: FeatureBase: Open-Source, Real-Time Database Built on Roaring Bitmaps

#13

Post by kordlessagain » Wed, Sep 14, 2022, 9:03 PM UTC

wentejing wrote: post #6

Why is the bigmap database faster than other distributed database? and what's the differences?

Instead of storing values, like "dog", "cat", or "mouse" it stores (in this example) three binary numbers:

000 - whatever needs to associate with animals, but has no associations currently

001 - whatever it is is associated with having a "mouse" included

111 - whatever it is is associated with having a "dog", a "cat" and a "mouse" included

In the past, high cardinality data sets weren't good for storing in binary form, or a binary index, but nowadays there are ways around this. So, that list of animals could be quite large.

The primary reason it's so much faster is that many CPUs nowadays can do 10s of lookups in a single instruction cycle. That makes them extremely fast.

Re: FeatureBase: Open-Source, Real-Time Database Built on Roaring Bitmaps

#14

Post by deniscoady » Thu, Sep 15, 2022, 12:43 AM UTC

JAA1337 wrote: post #8

kurosknight wrote: post #7

Earlier quoted context omitted.

They are useful for categorical variables. For example, is a record in the "Likes motorcycles" category? They are fast because (well, one reason) bitwise logical operations are very fast for CPUs to do. Adtech is an example of a sector that benefits from this...they slice and dice datasets a lot to target ad campaigns and such. Being able to do that quickly is useful.

So are you saying that the data is stored in categories which allows for those types of lookups to run faster? Do you have specifics on how the design of a bitmap based database achieves this? How does it maintain these relationships? Just through 0 and 1's? I guess it's easy for me to visualize both row and column based. Im struggling with the bitmaps concept.

No post body was provided.