Binary vector embeddings are so cool
emschwartz.me
Binary vector embeddings are so cool
1–10 of 13 posts
Re: Binary vector embeddings are so cool
#2That is crazy. So how do you use these right now? Do you store vectors on disk and iterate over them, or something else?
How do you do the search?
Re: Binary vector embeddings are so cool
#3[deleted]
Re: Binary vector embeddings are so cool
#4That is crazy. So how do you use these right now? Do you store vectors on disk and iterate over them, or something else? How do you do the search?
Yeah, that's exactly right. You take the embedding of your query and then compare it to the embeddings of your documents (or chunks of documents).
You can use vector databases that implement the Approximate Nearest Neighbors (ANN) algorithm, or if you don't have too many documents you can just brute force the similarity comparison.
I'm not sure at what point you _really_ need or want a vector database, but anecdotally, calculating the Hamming distance for a couple thousand vectors seems pretty negligible.
Re: Binary vector embeddings are so cool
#5It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do something like randomly flip bits with a probability weighted by the severity of the error during backprop… anyway, I’m sure there’s plenty of literature about this.
Re: Binary vector embeddings are so cool
#6That is crazy. So how do you use these right now? Do you store vectors on disk and iterate over them, or something else? How do you do the search?
Since hamming distance calculations take relatively few gates, its might be possible to compute those in-memory(ram/flash) to achieve massive amount of paralellism(for ex, a specific hamming distance calculations block for each stored vector) and at much lower power.
This could be useful for web-scale embeddings search.
Re: Binary vector embeddings are so cool
#7It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do something like randomly flip bits with a probability weighted by the severity of the error during backprop… anyway, I’m sure there’s plenty of literature about this.
You may find this interesting:
https://writings.stephenwolfram.com/2024/08/whats-really-goi...
Re: Binary vector embeddings are so cool
#8It got me thinking, what might it look like to natively train a binary quantized embedding model? You can’t do calculus per se on {0,1}, but maybe you could do something like randomly flip bits with a probability weighted by the severity of the error during backprop… anyway, I’m sure there’s plenty of literature about this.
Probably just an extreme version of quantization-aware training? During training you round the prediction to the range you want, but keep it as a float.
Since rounding isn’t differentiable there’s fancy techniques to approximate that as well.
> QAT backward pass typically uses straight-through estimators (STE), a mechanism to estimate the gradients flowing through non-smooth functions
Re: Binary vector embeddings are so cool
#9Is it fair to say that Hamming distance relates to cosine similarity in this context like ReLU relates to the Sigmoid function for neural networks?
Re: Binary vector embeddings are so cool
#10Yes, binary embeddings are cool indeed. And because they can be so small they even double down as powerful cluster identifiers. Built this a while ago: https://huggingface.co/spaces/iscc/iscc-sct