Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

github.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#21

Post by ogrisel »

ogrisel wrote: post #19
orestis wrote: post #15

Are there examples on how this can be used for topic modeling, document similarity etc? All the examples I’ve seen (gensim) use bag-of-words which seems to be outdated.

Big transformers neural network are probably overkill for topic modeling. More traditional methods implemented in Gensim or scikit learn such as tfidf vectors followed by SVD (aka LSI) or LDA or NMF are probably just fine to extract topics (soft clustering).

The reason is that you do not need to finely understand the structure of individual sentences to group documents by similar topics. Word order does not matter much for this task. Hence the success of methods that use Bag of Words (eg TFIDF) as their input representation.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#22

Post by visarga »

LunaSea wrote: post #20
brockf wrote: post #11

Earlier quoted context omitted.

Most implementations are actually moving in the opposite direction. Previously, there was a tendency to look to aggregate words into phrases to better capture the "context" of a word. Now, most approaches are splitting words into sub-word parts or even characters. With networks that capture temporal relationships across tokens (as opposed to older, "bag of words" models), multi-word patterns can effectively be captur…

> multi-word patterns can effectively be captured by attending to the temporal order of sub-word parts Indeed. Do you have an example of a library or snippet that demonstrates this? My limited understanding of BERT (and other) word embeddings was that they only contain the word's position in the 728 (I believe) dimensional space but doesn't contain queryable temporal information no? I like ngrams as a sort of untagge…

> Do you have an example of a library or snippet that demonstrates this?

All NLP neural nets (based on LSTM or Transformer) do this. It's their main function - to create contextual representations of the input tokens.

The word 'position' in the 728 dimensional space is an embedding and it can be compared with other words by dot product. There are libraries that can do dot product ranking fast (such as annoy).

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#23

Post by visarga »

echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

> What problems can you solve with NLP?

It's mostly understanding text and generating text. You can do named entity extraction, question answering, summarisation, dialogue bots, information extraction from semi-structured documents such as tables and invoices, spelling correction, typing auto-suggestions, document classification and clustering, topic discovery, part of speech tagging, syntactic trees, language modelling, image description and image question answering, entailment detection (if two affirmations support one another), coreference resolution, entity linking, intent detection and slot filling, build large knowledge bases (databases of triplets subject-relation-object), spam detection, toxic message detection, ranking search results in search engines and many many more.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#24

Post by PeterisP »

LunaSea wrote: post #20
brockf wrote: post #11

Earlier quoted context omitted.

Most implementations are actually moving in the opposite direction. Previously, there was a tendency to look to aggregate words into phrases to better capture the "context" of a word. Now, most approaches are splitting words into sub-word parts or even characters. With networks that capture temporal relationships across tokens (as opposed to older, "bag of words" models), multi-word patterns can effectively be captur…

> multi-word patterns can effectively be captured by attending to the temporal order of sub-word parts Indeed. Do you have an example of a library or snippet that demonstrates this? My limited understanding of BERT (and other) word embeddings was that they only contain the word's position in the 728 (I believe) dimensional space but doesn't contain queryable temporal information no? I like ngrams as a sort of untagge…

When using BERT (and all the many things like it, such as earlier ELMO, ULMfit and later ROBERTA/ERNIE/ALBERTa/etc) as the 'embeddings' you provide as input all the tokens in a sequence. You don't get an "embedding for word foobar in position 123", you get an embedding for all the sequence at once, so whatever corresponds to that token is a 728-dimensional "embedding for word foobar in position 123 conditional on all the particular other words that were before and after it'. Including very long-distance relations.

One of the simpler ways to try that out in your code seems to be running BERT-as-a-service https://github.com/hanxiao/bert-as-service , or alternatively the huggingface libraries that are discussed in the original article.

It's kind of the other way around compared to word2vec-style systems; before that you used to have a 'thin' embedding layer that's essentially just a lookup table followed by a bunch of complex layers of neural networks (e.g. multiple Bi-LSTMs followed by CRF); in the 'current style' you have "thick embeddings" which is running through all the many transformer layers in a pretrained BERT-like system, followed by a thin custom layer that's often just glorified linear regression.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#25

Post by brokensegue »

mraison wrote: post #12
echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

I believe many folks are particularly attracted to NLP because the Turing test [1] is an NLP problem. [1] https://en.m.wikipedia.org/wiki/Turing_test

Disagree. I think it's value is mostly unrelated to that.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#26

Post by orestis »

ogrisel wrote: post #21
ogrisel wrote: post #19

Earlier quoted context omitted.

Big transformers neural network are probably overkill for topic modeling. More traditional methods implemented in Gensim or scikit learn such as tfidf vectors followed by SVD (aka LSI) or LDA or NMF are probably just fine to extract topics (soft clustering).

The reason is that you do not need to finely understand the structure of individual sentences to group documents by similar topics. Word order does not matter much for this task. Hence the success of methods that use Bag of Words (eg TFIDF) as their input representation.

It might be that the corpus I was trying to cluster needs better preprocessing, or perhaps better n-grams. Using Bigrams only I saw a lot of common words that were meaningless, but adding them as stop words made the results worse. Hence my wondering if some other vectorization would produce better results.

On a related note, as a newcomer just trying to get things done (i.e. applied NLP) I find the whole ecosystem great but frustrating, so many frameworks and libraries but not clear ways to compose them together. Any resources out there that help make a sense of things?

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#29

Post by manojlds »

screye wrote: post #17

I can't believe the level of productivity this Hugging face team has. They seemed to have found the ideal balance of software engineering capability and Neural network knowledge, in a team of highly effective and efficient employees. Idk what their monetization plan is as a startup, but it is 100% undervalued at 20 million, and that is just the quality of that team. Now, if only I can figure out how to put a few thou…

> Idk what their monetization plan is as a startup

> put a few thousand $ in a series-A

Not a good idea.

Post reply on HN