[deleted]
Vectors are over, hashes are the future of AI
11–20 of 27 posts
Re: Vectors are over, hashes are the future of AI
#12I feel like I have been talking about LSH for years
I recall Uber using it in a white paper writeup. Is the speedup that remarkable? I'd be curious at to the increase in speed versus loss of precision.
I view ANN search for retrieval as clear winner in retrieval methods for domains with very large number of items like O(10 million +) that you want to high amount searches. I've worked at a couple different major tech companies and I'd consider two tower models + ANN as a classic pair. One tower for request embedding called on every request. One tower for item embedding. Compute all item embeddings periodically and build an ANN index. The top dot product of the two with minor constraints can be done by just running request tower and then doing search in the index.
The speed up is really necessity as direct methods are just too expensive both dollar wise and time wise (lowish latency is goal).
Re: Vectors are over, hashes are the future of AI
#13Whoever wrote the article must have done a cursory search at best, I'm surprised they didn't mention semantic hashing by Salakhutdinov & Hinton (2007) https://www.cs.utoronto.ca/~rsalakhu/papers/semantic_final.p... Edit: also, talking about LSH, must check out FAISS library https://github.com/facebookresearch/faiss and the current SOTA http://ann-benchmarks.com/
Interesting. Hadn’t seen it actually. BOW is problematic as a starting point, but this is neat and a long time ago too.
Re: Vectors are over, hashes are the future of AI
#14I like to speculate for reasons this might or might not make sense at several levels, although mostly just conjecturing. The fact everything works is very interesting, but it seems so hard to come up with something concrete.
You have a map from some high dimensional vector space ~ k^N -> H, some space of hashes. H sort of looks one dimensional. I assume that actually the interesting geometry of your training data lies on a relatively low dimensional subvariety/subset in k^N, so maybe its not actually that bad? It could be a really twisted and complicated curve.
However you still need to somehow preserve the relative structure right? Things that are far apart in k^N need to be far apart in H. Seems like you want things to at least approximately be an isometry. Although there are things like space filling curves that might do this for some degree.
Also maybe even though H looks low dimensional, it can actually capture quite a bit (if your data is encoded as a coefficient of a power of 2, you could think of powers of 2 as some sort of basis, so maybe it is also pretty high dimensional).
Re: Vectors are over, hashes are the future of AI
#15I feel like I have been talking about LSH for years
I recall Uber using it in a white paper writeup. Is the speedup that remarkable? I'd be curious at to the increase in speed versus loss of precision.
Yes. Storage also. You can get >99% ordering quality of exhaustive cosine with a tiny fraction of memory usage
Re: Vectors are over, hashes are the future of AI
#16hm. I'd like to believe, but the arguments here seem a bit obtuse. No one measures vector distance using the hamming distance on binary representations. That's silly. We use L1 or L2, usually, and the binary encoding of the numbers is irrelevant. It sounds like the LSH is maaaaybe equivalent to vector quantization. In which case this would be a form of regularization, which sometimes works well, and sometimes meh.
Goal is to end up with binary encoding so that Hamming distances approximate Euclidean nearest neighbors (so basically L2). Combine with quantized SIFT/GIST for images and you end up with a fast-and-dirty model with decent results.
Re: Vectors are over, hashes are the future of AI
#17This idea goes back to "sparse distributed memory" developed by NASA research in the 80s. It's a content-addressable memory where content hashes are encoded & decoded by neural network and similar items are in proximity to each other in the embedding space, and similarity measured via Hamming distance. https://en.wikipedia.org/wiki/Sparse_distributed_memory
Re: Vectors are over, hashes are the future of AI
#18hm. I'd like to believe, but the arguments here seem a bit obtuse. No one measures vector distance using the hamming distance on binary representations. That's silly. We use L1 or L2, usually, and the binary encoding of the numbers is irrelevant. It sounds like the LSH is maaaaybe equivalent to vector quantization. In which case this would be a form of regularization, which sometimes works well, and sometimes meh.
I was part of the above article. Happy to answer questions.
In terms of accuracy, it totally depends on the resolution needed. We can get >99% accuracy of L2 waaaaay faster with 1/10 of the memory overhead. For what we are doing that is the perfect trade off.
In terms of LSH, we tried projection hashing and quantization and were always disappointed.
Re: Vectors are over, hashes are the future of AI
#19hm. I'd like to believe, but the arguments here seem a bit obtuse. No one measures vector distance using the hamming distance on binary representations. That's silly. We use L1 or L2, usually, and the binary encoding of the numbers is irrelevant. It sounds like the LSH is maaaaybe equivalent to vector quantization. In which case this would be a form of regularization, which sometimes works well, and sometimes meh.
I was part of the above article. Happy to answer questions. In terms of accuracy, it totally depends on the resolution needed. We can get >99% accuracy of L2 waaaaay faster with 1/10 of the memory overhead. For what we are doing that is the perfect trade off. In terms of LSH, we tried projection hashing and quantization and were always disappointed.
Hello! First I would like to say this is a very cool writeup. I'm not a computer scientist but do dabble a bit in neural networks. Is it possible this could be used to build a convolutional neural network?
Re: Vectors are over, hashes are the future of AI
#20Hi, my interest got piqued. I'm developing a similarity feature where I compare embeddings of a sentence and its translation. I wanted to know if the hashing method would be faster that the pytorch multiplication by which I get the sentence similarities. Going from strings to bytes, hashing and comparing is very fast. But if I get the embeddings, turn them into bytes, hash them and compare them, both methods take almost the same time.
I used this Python library: https://github.com/trendmicro/tlsh.