Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Word2Bits – Quantized Word Vectors

github.com

Post reply on HN
Print view View original

101–110 of 113 posts

Re: Show HN: Word2Bits – Quantized Word Vectors

#101

Post by dougb5 »

rpedela wrote: post #92
dougb5 wrote: post #38

This is great and echoes a recent fascination for me. One application of compact word embeddings is that if they're small enough you can ship a whole model to the user in a web app so that semantic computations can be done entirely client-side, which is useful for privacy. I did a naive 1-bit quantization a few months ago in order to fit a large-vocabulary word embedding into a smallish ( https://docs.google.com/pres…

Can you expand on the privacy use case? I don't understand how applying a word vector model on the client improves privacy.

The tool I made works with local data -- it analyzes your Google search history from a file on your disk. You can filter it by specifying a topic, which is then matched semantically against the search queries in your history using the word vectors. Given the sensitive nature of this data, I felt it important not to send these topic queries (nor any of the raw search history queries) to a remote server. I can also imagine using the word vectors to do a topic clustering of the search history.

Re: Show HN: Word2Bits – Quantized Word Vectors

#102

Post by nl »

creichenbach wrote: post #68
nl wrote: post #51

Earlier quoted context omitted.

FastText can generate word vectors for partial and/or unknown words based on similar chargram patterns.

That sounds interesting, do you happen to have a documentation link or similar? I can't seem to find any info about it.

There are some somewhat reasonable docs at the bottom of https://fasttext.cc/docs/en/unsupervised-tutorial.html

I really should do a blog post about it or something.

Re: Show HN: Word2Bits – Quantized Word Vectors

#103

Post by rspeer »

fvdessen wrote: post #99
rspeer wrote: post #95

Earlier quoted context omitted.

You've given an example of where a human decision in a data-processing pipeline harms the result. There are many examples where human decisions are a benefit. Look at every other decision in a successful speech-recognition pipeline for examples. We're not just putting raw waveforms into a black box handed to us by aliens. If you read the paper, you'll see that de-biasing is not based on assumptions, it is also based…

I read the Bolukbasi paper and I now see where you are getting at. I think that the problem is not one of bias but rather that words vector embed the whole meaning regardless of the context. And that debiasing is rather stop-gap solution as there is an endless list of bias to correct depending on the usage of the model. You may have fixed PROGRAMMER - MAN + WOMAN = HOMEMAKER, and still get SOLDIER - AMERICAN + ARAB =…

Yep, so there's a lot of work to do. Bolukbasi, Cheng, et al. will straightforwardly admit that their model only applies to the one axis of gender bias. Fortunately there are other people working on this too, and I am one of them.

The baked-in assumption that Arabs or Muslims are terrorists, or that terrorists are Arabs or Muslims, is something that the de-biasing process in ConceptNet Numberbatch (which I make) attempts to mitigate at the same time as gender and racial bias. And of course there is much more to do.

It's a fascinating and productive field of research. Why did you have such a negative initial reaction to it?

Re: Show HN: Word2Bits – Quantized Word Vectors

#104

Post by sushirain »

creichenbach wrote: post #66
opportune wrote: post #40

Earlier quoted context omitted.

You could modify word2vec to embed word fragments as part of the training process. You could also use stemming before training, or if you have a decent amount of computational resources you could embed trie entries with word2vec representations of word fragments and probabilistic models of the likely next character/syllable/word, which would allow you to use something like a markov process For word completion, I woul…

Our problem is not about auto-completion (we're not dealing with that much data to need sophisticated algorithms for that). What we're doing with our NN is ordering the set of results (matches) we already have. In other words, we're assigning a relevance number in [0, 1] to each result, based on the query string and training based on past user choices (clicking a result). In order to maintain some consistency and rob…

You can use a character RNN that reads the fragmented input char by char, and feed its output to your NN.

Re: Show HN: Word2Bits – Quantized Word Vectors

#105

Post by maxlam »

yorwba wrote: post #74
maxlam wrote: post #13

Earlier quoted context omitted.

You're definitely right, the quantization function and its values definitely have an impact on performance. For 1 bit I think I tried something like -1/+1, -.5/+.5, -.25/+.25, -.333/+.333. and something like -10/+10 -- (and I think a few more). It seemed -.333/+.333 worked the best while +10/-10 did the worst on the google analogy task (getting like 0% right). All this was tuned on 100MB of Wikipedia data.

Have you considered doing gradient descent on the quantization steps? It looks to me like the model should be differentiable with respect to those values, so I'm not sure why you'd have to fix them to a constant.

Hm what do you mean? I'm not quite seeing how to differentiate with respect to the quantization steps.

Re: Show HN: Word2Bits – Quantized Word Vectors

#106

Post by maxlam »

Maybestring wrote: post #88
maxlam wrote: post #25

Earlier quoted context omitted.

Definitely tried to figure out if the dimensions mean anything -- as far as I can tell they don't really mean much :(

If you want them to be meaningful without changing the model... Couldn't you rotate the basis to minimize the distance between each basis vector and it's nearest neighbor?

Haven't tried this but this is definitely a good idea for visualizing what's going on!

Re: Show HN: Word2Bits – Quantized Word Vectors

#107

Post by fvdessen »

rspeer wrote: post #103
fvdessen wrote: post #99

Earlier quoted context omitted.

I read the Bolukbasi paper and I now see where you are getting at. I think that the problem is not one of bias but rather that words vector embed the whole meaning regardless of the context. And that debiasing is rather stop-gap solution as there is an endless list of bias to correct depending on the usage of the model. You may have fixed PROGRAMMER - MAN + WOMAN = HOMEMAKER, and still get SOLDIER - AMERICAN + ARAB =…

Yep, so there's a lot of work to do. Bolukbasi, Cheng, et al. will straightforwardly admit that their model only applies to the one axis of gender bias. Fortunately there are other people working on this too, and I am one of them. The baked-in assumption that Arabs or Muslims are terrorists, or that terrorists are Arabs or Muslims, is something that the de-biasing process in ConceptNet Numberbatch (which I make) atte…

Because I find in AI's candid responses to the questions we ask a fascinating opportunity to understand humanity better. When we try to conform those answers to our desired biases we learn less from it. But now I understand that such aspects are a subset of the greater problem of getting our models to answer in a specific context.

Ultimately the Model would need to detect & learn the contexts by itself.

And more to the point; I see you are publishing data sets after correcting them for fairness. I think the unfair results can sometimes be more useful. I once prototyped a short-story tagging & recommendation system based on word2vec and I would not be surprised if raw vectors gave better results. People's taste in litterature (especially romantic) are very dependant on gender stereotypes.

Re: Show HN: Word2Bits – Quantized Word Vectors

#109

Post by yorwba »

maxlam wrote: post #105
yorwba wrote: post #74

Earlier quoted context omitted.

Have you considered doing gradient descent on the quantization steps? It looks to me like the model should be differentiable with respect to those values, so I'm not sure why you'd have to fix them to a constant.

Hm what do you mean? I'm not quite seeing how to differentiate with respect to the quantization steps.

Say you have a function f(q(x)) where q quantizes x into one of s_1, ..., s_n. Then if q(x) = s_i for a certain x, df/ds_i = df/dq and df/ds_j = 0 for all j != i.

That breaks down for values of x precisely at the boundary between steps, so I should have qualified "differentiable" with "almost everywhere".

It also occurs to me that this might interact strangely with the approximation dq/dx = 1, but since the quantization steps are globally shared, I think it should be stable anyway.

If the evaluation suite for your code doesn't require too much manual interaction, I might try and see for myself.

Re: Show HN: Word2Bits – Quantized Word Vectors

#110

Post by _mhr_ »

loxias wrote: post #75

Interesting. So, this approach computes a "traditional" neural embedding, in say, R^50, and then "brutally" replaces each of the reals with an integer in Z_2,4,8... I can't quite put my finger on it, but my hunch is that this naive method, while already delivering interesting results, can be drastically improved upon. * don't use a fixed bit depth for all vector components I guess it depends on what you're trying to…

I would appreciate if you elaborated on your supposed improvements.
Post reply on HN