Show HN: Word2Bits – Quantized Word Vectors
1–10 of 113 posts
Re: Show HN: Word2Bits – Quantized Word Vectors
#2Can someone explain how to read the graph?
Re: Show HN: Word2Bits – Quantized Word Vectors
#3Can someone explain how to read the graph?
From the paper
> Figure 2 shows a visualisation of 800 dimensional 1 bit word vectors trained on English Wikipedia (2017). The top 100 closest and furthest word vectors to the target word vector are plotted. Distance is measured by dot product; every 5 word vectors are labelled. A turquoise line separates the 100 closest vectors to the target word from the 100 furthest vectors (labelled “...”). We see that there are qualitative similarities between word vectors whose words are related to each other.
Re: Show HN: Word2Bits – Quantized Word Vectors
#4Can someone explain how to read the graph?
Yeah, I should definitely put more detail in the writeup -- thanks for the feedback!
What's happening with figure 1a (epochs vs google accuracy) is that as you train for more epochs the full precision loss continues to decrease (dotted red line) but accuracy also starts decreasing (solid red line). This indicates overfitting (since you'd expect accuracy to increase if loss decreases). The blue lines (quantized training with 1 bit) do not show this which suggests that quantized training seems to act as a form of regularization.
Figure 1b is pretty similar, except on the x axis we have vector dimension. As you increase vector dimension, full precision loss decreases, yet after a certain point full precision accuracy decreases as well. I took this to mean that word2vec training was overfitting with respect to vector dimension.
Re: Show HN: Word2Bits – Quantized Word Vectors
#5Can someone explain how to read the graph?
Yeah, I should definitely put more detail in the writeup -- thanks for the feedback! What's happening with figure 1a (epochs vs google accuracy) is that as you train for more epochs the full precision loss continues to decrease (dotted red line) but accuracy also starts decreasing (solid red line). This indicates overfitting (since you'd expect accuracy to increase if loss decreases). The blue lines (quantized traini…
Oops, thought you meant the graphs (with the dotted/solid lines) in the writeup.
If you're referring to the image under "Visualizing Quantized Word Vectors" then each row is a word vector (and there are only two colors since each parameter is either -1/3 or +1/3).
Re: Show HN: Word2Bits – Quantized Word Vectors
#6The data engineer in me thanks you. Great work.
The paper would be a great weekend read.
Re: Show HN: Word2Bits – Quantized Word Vectors
#7interesting that 'artists' is a furthest neighbor of 'man' in the example
Re: Show HN: Word2Bits – Quantized Word Vectors
#8interesting that 'artists' is a furthest neighbor of 'man' in the example
This might be because "Artist" has an uppercase "A" -- I trained all the word vectors to be case sensitive so "Artist" is not the same as "artist" (which should be closer to "man" than "Artist")
Re: Show HN: Word2Bits – Quantized Word Vectors
#9interesting that 'artists' is a furthest neighbor of 'man' in the example
This might be because "Artist" has an uppercase "A" -- I trained all the word vectors to be case sensitive so "Artist" is not the same as "artist" (which should be closer to "man" than "Artist")
Why would you do that? If you look at something like LSA, they goal is to uniform those, rather than distinguish. artist and Artist should be (near)100% match; what are you trying to do here?
Re: Show HN: Word2Bits – Quantized Word Vectors
#10Earlier quoted context omitted.
This might be because "Artist" has an uppercase "A" -- I trained all the word vectors to be case sensitive so "Artist" is not the same as "artist" (which should be closer to "man" than "Artist")
Why would you do that? If you look at something like LSA, they goal is to uniform those, rather than distinguish. artist and Artist should be (near)100% match; what are you trying to do here?
Main reason I did it this way is because Facebook's DrQA (which I evaluate the vectors on for the SQuAD task) uses case sensitive vectors.
Was a tough decision between choosing whether to train case sensitive vectors / case insensitive vectors and a future task would be to train case-insensitive vectors.