Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Word2Bits – Quantized Word Vectors

github.com

Post reply on HN
Print view View original

41–50 of 113 posts

Re: Show HN: Word2Bits – Quantized Word Vectors

#41

Post by hnuser1234 »

rspeer wrote: post #26
maxlam wrote: post #23

Earlier quoted context omitted.

Interesting, definitely need to think about debiasing. Seems like it won't really work straight out of the box since it'd destroy the 1 bit-ness of the vectors. Though if only a few of the vectors are de-biased then you can still save a lot of space since all the other vectors are still represented using 2 numbers (while the de-biased vectors are represented using the full range of 32 bit numbers).

There may be a way to build it into the loss function so that it happens before the quantization, right? (and holy crap, look how fast the HN conservatives are getting to my comment)

Literally everything is either Left or Right. Anyone who disagrees with you is definitely a sexist conservative. No other possibility. The center is a black hole.

Re: Show HN: Word2Bits – Quantized Word Vectors

#42

Post by [deleted] »

rspeer wrote: post #30
maxlam wrote: post #28

Earlier quoted context omitted.

Hm not sure, would need to think more about this -- definitely an interesting idea though!

Thanks for being willing to discuss it. And sorry if I'm complicating your task. But any interesting release of NLP data has the potential to affect the way the field progresses, so take it as a compliment that I consider this an interesting release of NLP data. That's why I'm asking you to actively consider the downstream effects of word vectors and find out if you can make them better.

[deleted]

Re: Show HN: Word2Bits – Quantized Word Vectors

#43

Post by maxlam »

opportune wrote: post #33
Sukotto wrote: post #14

I thought I knew what a vector was, but I have no idea what any of this project means. Would someone please explain in simple language what this is, and why it's cool?

Quick run down on word2vec and what happened here: A big problem with NLP is understanding the semantic associations between words (or even lemmas. Lemmas in this context refer to different meanings of the same word, like a baseball bat vs. a vampire bat ). For example "run" and "sprint" are similar in meaning but convey different connotations; kings and queens are both high-level monarchs but we need to encode the d…

Training these quantized word vectors has to be done in full precision (so no memory gains during training the word vectors).

But when you save them to disk every value is either -1/3 or +1/3 so one could encode the word vectors in binary. This can lead to reducing memory usage during application time if you kept the word vectors in this compressed format (though you'd need to write a decode function in tensorflow or pytorch to take a sequence of bits corresponding to a word and convert it into a vector of -1/3s and +1/3s)

Re: Show HN: Word2Bits – Quantized Word Vectors

#44

Post by hnuser1234 »

rspeer wrote: post #17

It's interesting and slightly uncomfortable that the illustration for similar words uses the word "man" as an example, given the gender biases that result from learning word vectors solely from word distributions. To explain, although I’m sure the author himself is familiar with the issue: For any word that is disproportionately associated with one gender in the corpus, the model will learn that gender difference as…

One of the closest related words is lady, and then effeminate, so it's quite unclear exactly what there is to be upset about? In fact it appears it's saying men are related to being effeminate, which should get you excited.

Re: Show HN: Word2Bits – Quantized Word Vectors

#45

Post by Jack000 »

can this be done with fasttext as well?

Word2bits is definitely great for memory-constrained applications but for server use memory isn't as much a constraint (there's a direct word -> vector relationship so you can just put it in a database)

it would be amazing to combine this with fasttext's ability to generate vectors for out-of-vocab words.

Re: Show HN: Word2Bits – Quantized Word Vectors

#46

Post by igravious »

creichenbach wrote: post #36
wodenokoto wrote: post #24

Earlier quoted context omitted.

> To our surprise, we couldn't find any example or description of someone doing this before. Is this such an uncommon problem or did we just not search in the right places? This is one of the defining differences between Word2Vec and Fasttext. But fasttext incorporates these character vectors as part of calculating the semantic vector, so you can't expect carg and cargo to end up being similar, but people have though…

Yeah, Levenstein distance is pretty close to our goal metric of "similarity". The thing is that we're feeding the search query into a neural network, hence we need some kind of vector represenation.

Need?

Or don't feed the search query directly into a neural network?

Re: Show HN: Word2Bits – Quantized Word Vectors

#47

Post by visarga »

dougb5 wrote: post #38

This is great and echoes a recent fascination for me. One application of compact word embeddings is that if they're small enough you can ship a whole model to the user in a web app so that semantic computations can be done entirely client-side, which is useful for privacy. I did a naive 1-bit quantization a few months ago in order to fit a large-vocabulary word embedding into a smallish ( https://docs.google.com/pres…

You can also replace rare words with a linear combination of 1..3 of their closest neighbours, especially for topic classification tasks. This would allow higher precision for more frequent words and still handle the rare words, reducing the number of vectors you need to send to the thin client.

Re: Show HN: Word2Bits – Quantized Word Vectors

#48

Post by opportune »

maxlam wrote: post #43
opportune wrote: post #33

Earlier quoted context omitted.

Quick run down on word2vec and what happened here: A big problem with NLP is understanding the semantic associations between words (or even lemmas. Lemmas in this context refer to different meanings of the same word, like a baseball bat vs. a vampire bat ). For example "run" and "sprint" are similar in meaning but convey different connotations; kings and queens are both high-level monarchs but we need to encode the d…

Training these quantized word vectors has to be done in full precision (so no memory gains during training the word vectors). But when you save them to disk every value is either -1/3 or +1/3 so one could encode the word vectors in binary. This can lead to reducing memory usage during application time if you kept the word vectors in this compressed format (though you'd need to write a decode function in tensorflow or…

Oh interesting I see, so this is like a digital format mostly for sharing models between storage / over networks. I definitely think it would be possible (and useful!) to extend it to in-memory usage, though a C-function wrapper might be better than a native python function. Personally I'm often more frustrated by word2vec's size in memory than in storage so it might be used more in this manner. Would you mind if I submit a pull request?

Re: Show HN: Word2Bits – Quantized Word Vectors

#49

Post by strawcomb »

hnuser1234 wrote: post #44
rspeer wrote: post #17

It's interesting and slightly uncomfortable that the illustration for similar words uses the word "man" as an example, given the gender biases that result from learning word vectors solely from word distributions. To explain, although I’m sure the author himself is familiar with the issue: For any word that is disproportionately associated with one gender in the corpus, the model will learn that gender difference as…

One of the closest related words is lady, and then effeminate, so it's quite unclear exactly what there is to be upset about? In fact it appears it's saying men are related to being effeminate, which should get you excited.

What a great quantitative evaluation of bias you just carried out.

Re: Show HN: Word2Bits – Quantized Word Vectors

#50

Post by jacobush »

opportune wrote: post #33
Sukotto wrote: post #14

I thought I knew what a vector was, but I have no idea what any of this project means. Would someone please explain in simple language what this is, and why it's cool?

Quick run down on word2vec and what happened here: A big problem with NLP is understanding the semantic associations between words (or even lemmas. Lemmas in this context refer to different meanings of the same word, like a baseball bat vs. a vampire bat ). For example "run" and "sprint" are similar in meaning but convey different connotations; kings and queens are both high-level monarchs but we need to encode the d…

I wonder, shouldn't the vector space ideally be much more than high-dimensional than hundreds? Like many thousands at least. (I mean, our brains likely can hold much more than hundreds of dimensions internally, I'd guess.) It's just that we don't have the computing power yet?
Post reply on HN