Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

github.com

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#41

Post by dzink »

mark_l_watson wrote: post #10

I love the work done and made freely available by both spaCy and HuggingFace. I had my own NLP libraries for about 20 years, simple ones were examples in my books, and more complex and not so understandable ones I sold as products and pulled in lots of consulting work with. I have completely given up my own work developing NLP tools, and generally I use the Python bindings (via the Hy language (hylang) which is a Lis…

Very interested in working on symbolic and deep learning projects for NLP as well.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#42

Post by oddnearfuture »

ogrisel wrote: post #19
orestis wrote: post #15

Are there examples on how this can be used for topic modeling, document similarity etc? All the examples I’ve seen (gensim) use bag-of-words which seems to be outdated.

Big transformers neural network are probably overkill for topic modeling. More traditional methods implemented in Gensim or scikit learn such as tfidf vectors followed by SVD (aka LSI) or LDA or NMF are probably just fine to extract topics (soft clustering).

With the appropriate amount of data of course.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#44

Post by Erlich_Bachman »

PeterisP wrote: post #24
LunaSea wrote: post #20

Earlier quoted context omitted.

> multi-word patterns can effectively be captured by attending to the temporal order of sub-word parts Indeed. Do you have an example of a library or snippet that demonstrates this? My limited understanding of BERT (and other) word embeddings was that they only contain the word's position in the 728 (I believe) dimensional space but doesn't contain queryable temporal information no? I like ngrams as a sort of untagge…

When using BERT (and all the many things like it, such as earlier ELMO, ULMfit and later ROBERTA/ERNIE/ALBERTa/etc) as the 'embeddings' you provide as input all the tokens in a sequence. You don't get an "embedding for word foobar in position 123", you get an embedding for all the sequence at once, so whatever corresponds to that token is a 728-dimensional "embedding for word foobar in position 123 conditional on all…

> in the 'current style' you have "thick embeddings" which is running through all the many transformer layers in a pretrained BERT-like system, followed by a thin custom layer that's often just glorified linear regression.

Would you say they are still usually called "embeddings" when using this new style? This sounds more like just a pretrained network which includes both some embedding scheme and a lot of learning on top of it, but maybe the word "embedding" stuck anyway?

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#45

Post by PeterisP »

Erlich_Bachman wrote: post #44
PeterisP wrote: post #24

Earlier quoted context omitted.

When using BERT (and all the many things like it, such as earlier ELMO, ULMfit and later ROBERTA/ERNIE/ALBERTa/etc) as the 'embeddings' you provide as input all the tokens in a sequence. You don't get an "embedding for word foobar in position 123", you get an embedding for all the sequence at once, so whatever corresponds to that token is a 728-dimensional "embedding for word foobar in position 123 conditional on all…

> in the 'current style' you have "thick embeddings" which is running through all the many transformer layers in a pretrained BERT-like system, followed by a thin custom layer that's often just glorified linear regression. Would you say they are still usually called "embeddings" when using this new style? This sounds more like just a pretrained network which includes both some embedding scheme and a lot of learning o…

They do seem to be still called "embeddings" although yes, that's become a somewhat misleading misonmer in some sense.

However, the analogy still is somewhat meaningful, because if you want to look at the properties of a particular word or token, it's not just a general pretrained network, it still preseves the one-to-one mapping between the input token and the output vector corresponding to each particular token; which is very important for all kinds of sequence labeling or span/boundary detection tasks. So you can use them just as word2vec embeddings - for example, if you'd do word similarity or word difference metrics with 'transformer-stack-embeddings' then that would work just as well as word2vec (though you'd have to get to a word-level measurement instead of wordpiece or BPE subword tokens) with the added bonus of having done contextual disambiguation; you probably could do a decent word sense disambiguation system just by directly clustering these embeddings; the mouse-as-animal and mouse-as-computer-peripheral should have clearly different embeddings.

Post reply on HN