Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Universal Sentence Encoder

arxiv.org

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Universal Sentence Encoder

#31

Post by slashcom »

nl wrote: post #12

Interesting. There's a big need for better vector representations of things in-between words (for which Word2Vec/Glove/FastText work well) and documents (which to me seems impossible. Yes I know about Doc2Vec etc, but really.. it works ok for paragraphs). Facebook's InferSent[1] has worked reasonably well for me for a variety of sentence level tasks, but I don't have anything I can point to to say that it is really s…

For the record, good old fashioned bag of words representations (tf-idf, LDA, LSA) still provide useful representations for documents. Obviously we hope to do better, but recently people act like there's no way of turning a document into a vector.

Re: Universal Sentence Encoder

#32

Post by tree_of_item »

laboo wrote: post #22
rahimnathwani wrote: post #10

Earlier quoted context omitted.

They made a way to take any sentence, and output a small array of numbers that represent its essence. You can use their model to find the essence of your own sentences. And then use it either directly (e.g. compare the essence of two sentences to see if they're saying roughly the same thing) or use it as a starting point for the model you need (e.g. if you're building a system to convert English sentences into French…

What do you mean by "its essence"? Is this a semantic essence?

The array of numbers represents some opaque statistical property of the sentence with respect to the others in the corpus the model was trained from. The hope is that this property will correlate with what we believe to be the sentence's meaning.

Re: Universal Sentence Encoder

#33

Post by thaumaturgy »

igravious wrote: post #6

“We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption…

I had been under the impression that you could just feed text into neural nets, and then ... magic!

But, no. As it turns out, the very first problem you encounter when trying to implement ML on text is that you need to transform the text into some set of numbers (the "vectors"), with the elements in the set matching the number of nodes in your input layer.

This is a tricky thing to do. You're essentially trying to "hash" the text in a way which uniquely represents the text you're working with and also gives the neural net something it can operate on. Which is to say, you can't just use a common hashing algorithm, because the neural net won't be able to learn anything from the random output of the hashing algorithm.

There are several different approaches being used for this. One of them, mentioned elsethread, is "bag-of-words", where you build a big dictionary of word-to-number associations and then do some variety of transformations on that. Another is "feature extraction", where you might try to input a value representing properties like the length of the sentence, the number of words, the vocabulary level of the words, and so on. (This would probably be a bad approach for most ML goals on long text.)

This paper presents another approach.

Re: Universal Sentence Encoder

#35

Post by nl »

slashcom wrote: post #31
nl wrote: post #12

Interesting. There's a big need for better vector representations of things in-between words (for which Word2Vec/Glove/FastText work well) and documents (which to me seems impossible. Yes I know about Doc2Vec etc, but really.. it works ok for paragraphs). Facebook's InferSent[1] has worked reasonably well for me for a variety of sentence level tasks, but I don't have anything I can point to to say that it is really s…

For the record, good old fashioned bag of words representations (tf-idf, LDA, LSA) still provide useful representations for documents. Obviously we hope to do better, but recently people act like there's no way of turning a document into a vector.

Bag of word representations work fine for some applications.

The reason people want better representations is for the applications where they don’t. For example, Bag of words doesn’t capture agreement or disagree well, whereas better representations can.

Re: Universal Sentence Encoder

#36

Post by mlevental »

sp821543 wrote: post #25
eruditepanda wrote: post #3

Earlier quoted context omitted.

It looks like an internal site, this is the link it is referring to: https://tfhub.dev/google/universal-sentence-encoder/1

https://www.tensorflow.org/hub/modules/google/universal-sent...

so does this work? am i getting redirected back to that page when i click the link because they're checking my user agent? i don't have tf installed on this machine in order to check but does getting the model through the tf api work?
Post reply on HN