Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

github.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#11

Post by brockf »

LunaSea wrote: post #8

It used to be that pre-DeepLearning tokenizers would extract ngrams (n-token sized chunks) but this doesn't seem to exist anymore in the word embedding tokenizers I've come by. Is this possible using HuggingFace (or another word embedding based library)? I know that there are some simple heuristics like merging noun token sequences together to extract ngrams but they are too simplistic and very error prone.

Most implementations are actually moving in the opposite direction. Previously, there was a tendency to look to aggregate words into phrases to better capture the "context" of a word. Now, most approaches are splitting words into sub-word parts or even characters. With networks that capture temporal relationships across tokens (as opposed to older, "bag of words" models), multi-word patterns can effectively be captured by attending to the temporal order of sub-word parts.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#12

Post by mraison »

echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

I believe many folks are particularly attracted to NLP because the Turing test [1] is an NLP problem.

[1] https://en.m.wikipedia.org/wiki/Turing_test

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#13

Post by useful »

Somewhat related, if someone want to build something awesome, I haven't seen anything that merges lucene with BPE/SentencePiece.

SentencePiece has to make it so you can shrink the memory requirements of your indexes for search and typeahead stuff.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#14

Post by starpilot »

echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

All of the above, it's like asking what problems can you solve with math? HuggingFace's transformers are said to be a swiss army knife for NLP. I haven't worked with them yet, but the main fundamental utility seems to be generating fixed-length vector representations of words. Word2vec started this, but the vectors have gotten much better with stuff like BERT.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#16

Post by Isn0gud »

starpilot wrote: post #14
echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

All of the above, it's like asking what problems can you solve with math? HuggingFace's transformers are said to be a swiss army knife for NLP. I haven't worked with them yet, but the main fundamental utility seems to be generating fixed-length vector representations of words. Word2vec started this, but the vectors have gotten much better with stuff like BERT.

I thought transformers are mainly used for multi-word embeddings?!

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#17

Post by screye »

I can't believe the level of productivity this Hugging face team has.

They seemed to have found the ideal balance of software engineering capability and Neural network knowledge, in a team of highly effective and efficient employees.

Idk what their monetization plan is as a startup, but it is 100% undervalued at 20 million, and that is just the quality of that team. Now, if only I can figure out how to put a few thousand $ in a series-A startup as just some guy.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#18

Post by crawdog »

echelon wrote: post #9

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP. What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation? What cool problems are there?

There's a lot! Sentence detection, parts of speech (POS) detection to name a couple. These can be used to determine key concepts in documents that lack metadata. For example: you could cluster on common phrases to identify relationships in data.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#19

Post by ogrisel »

orestis wrote: post #15

Are there examples on how this can be used for topic modeling, document similarity etc? All the examples I’ve seen (gensim) use bag-of-words which seems to be outdated.

Big transformers neural network are probably overkill for topic modeling. More traditional methods implemented in Gensim or scikit learn such as tfidf vectors followed by SVD (aka LSI) or LDA or NMF are probably just fine to extract topics (soft clustering).

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#20

Post by LunaSea »

brockf wrote: post #11
LunaSea wrote: post #8

It used to be that pre-DeepLearning tokenizers would extract ngrams (n-token sized chunks) but this doesn't seem to exist anymore in the word embedding tokenizers I've come by. Is this possible using HuggingFace (or another word embedding based library)? I know that there are some simple heuristics like merging noun token sequences together to extract ngrams but they are too simplistic and very error prone.

Most implementations are actually moving in the opposite direction. Previously, there was a tendency to look to aggregate words into phrases to better capture the "context" of a word. Now, most approaches are splitting words into sub-word parts or even characters. With networks that capture temporal relationships across tokens (as opposed to older, "bag of words" models), multi-word patterns can effectively be captur…

> multi-word patterns can effectively be captured by attending to the temporal order of sub-word parts

Indeed. Do you have an example of a library or snippet that demonstrates this?

My limited understanding of BERT (and other) word embeddings was that they only contain the word's position in the 728 (I believe) dimensional space but doesn't contain queryable temporal information no?

I like ngrams as a sort of untagged / unlabelled entity.

Post reply on HN