Exact binary vector search for RAG in 100 lines of Julia
1–10 of 25 posts
Re: Exact binary vector search for RAG in 100 lines of Julia
#2 for i in 0:7
c += (r >> i) & 1
end
Re: Exact binary vector search for RAG in 100 lines of Julia
#3for i in 0:7 c += (r >> i) & 1 end This is just popcnt, surely Julia has a built in for that.
From what I've heard it's actually faster to create a 256 byte lookup table than to use popcnt.
Re: Exact binary vector search for RAG in 100 lines of Julia
#4for i in 0:7 c += (r >> i) & 1 end This is just popcnt, surely Julia has a built in for that.
From what I've heard it's actually faster to create a 256 byte lookup table than to use popcnt.
It used to be pretty bad on old intel processors but nowadays it should be faster than an L1 fetch.
Re: Exact binary vector search for RAG in 100 lines of Julia
#5I'm not sure what is meant by "exact" here - do they describe their binarisation process at all? This seems more like an XOR benchmark than a rag benchmark, no mention of recall or other relevant performance metrics
Re: Exact binary vector search for RAG in 100 lines of Julia
#6Why not use the built in BitVector type that has specialized code for things like xor?
Re: Exact binary vector search for RAG in 100 lines of Julia
#7for i in 0:7 c += (r >> i) & 1 end This is just popcnt, surely Julia has a built in for that.
There is, it's called count_ones. Though I wouldn't be surprised if LLVM could maybe optimize some of these loops into a popcnt, but I'm sure it would be brittle
Re: Exact binary vector search for RAG in 100 lines of Julia
#8I'm not sure what is meant by "exact" here - do they describe their binarisation process at all? This seems more like an XOR benchmark than a rag benchmark, no mention of recall or other relevant performance metrics
Some (not all) of your questions may be answered by the linked article near the top of the submitted article, which goes into more detail about how much is lost quantizing to 1 bit (and 1 byte): https://huggingface.co/blog/embedding-quantization
Re: Exact binary vector search for RAG in 100 lines of Julia
#9For those like me who are not familiar with the field... The article assumes you know the entire context - as far as I could see there is no explanation of any part except the technical details.
RAG = Retrieval-Augmented Generation
The field is machine learning. Retrieval = get relevant documents. Generation = create answer for user (based on the docs).
Re: Exact binary vector search for RAG in 100 lines of Julia
#10for i in 0:7 c += (r >> i) & 1 end This is just popcnt, surely Julia has a built in for that.
author here. I thought there might be a machine instruction for this but wasn't sure, I also didn't know Julia had a count_ones that counted the 1s.
Thanks! With this the timings are even faster. I'll update the post.