Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines • Hacker News Board

Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#1

Post by julien_c » Mon, Jan 13, 2020, 4:40 PM UTC

Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines
github.com

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#2

Post by virtuous_signal » Mon, Jan 13, 2020, 4:45 PM UTC

I didn't realize that particular emoji had a name. I thought it was a play on this: https://en.wikipedia.org/wiki/Alien_(creature_in_Alien_franc...

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#3

Post by julien_c » Mon, Jan 13, 2020, 5:09 PM UTC

TL;DR: Hugging Face, the NLP research company known for its transformers library (DISCLAIMER: I work at Hugging Face), has just released a new open-source library for ultra-fast & versatile tokenization for NLP neural net models (i.e. converting strings in model input tensors).

Main features: - Encode 1GB in 20sec - Provide BPE/Byte-Level-BPE/WordPiece/SentencePiece... - Compute exhaustive set of outputs (offset mappings, attention masks, special token masks...) - Written in Rust with bindings for Python and node.js

Github repository and doc: https://github.com/huggingface/tokenizers/tree/master/tokeni...

To install: - Rust: https://crates.io/crates/tokenizers - Python: pip install tokenizers - Node: npm install tokenizers

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#4

Post by rsp1984 » Mon, Jan 13, 2020, 5:41 PM UTC

What does tokenization (of strings, I guess) do?

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#5

Post by wyldfire » Mon, Jan 13, 2020, 5:50 PM UTC

rsp1984 wrote: post #4

What does tokenization (of strings, I guess) do?

The README [1] shows a great example:

The sentence "Hello, y'all! How are you ?" is tokenized into words. Those words are then encoded into integers representative of the words' identity in the model's dictionary.

    >>> output = tokenizer.encode("Hello, y'all! How are you  ?")
    Encoding(num_tokens=13, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing, original_str, normalized_str])
    >>> print(output.ids, output.tokens, output.offsets)
    [101, 7592, 1010, 1061, 1005, 2035, 999, 2129, 2024, 2017, 100, 1029, 102]
    ['[CLS]', 'hello', ',', 'y', "'", 'all', '!', 'how', 'are', 'you', '[UNK]', '?', '[SEP]']
    [(0, 0), (0, 5), (5, 6), (7, 8), (8, 9), (9, 12), (12, 13), (14, 17), (18, 21), (22, 25), (26, 27), (28, 29), (0, 0)]

But there's also good detail in the source [2] which says, "A Tokenizer works as a pipeline, it processes some raw text as input and outputs an Encoding. The various steps of the pipeline are: ...."

[1] https://github.com/huggingface/tokenizers#quick-examples-usi...

[2] https://github.com/huggingface/tokenizers/tree/master/tokeni...

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#6

Post by hnaccy » Mon, Jan 13, 2020, 5:50 PM UTC

Great! Just did a quick test and got a 6-7x speedup on tokenization.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#7

Post by ZeroCool2u » Mon, Jan 13, 2020, 5:50 PM UTC

We use both SpaCy and HuggingFace at work. Is there a comparison of this vs SpaCy's tokenizer[1]?

1. https://spacy.io/usage/linguistic-features#tokenization

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#8

Post by LunaSea » Mon, Jan 13, 2020, 6:00 PM UTC

It used to be that pre-DeepLearning tokenizers would extract ngrams (n-token sized chunks) but this doesn't seem to exist anymore in the word embedding tokenizers I've come by.

Is this possible using HuggingFace (or another word embedding based library)?

I know that there are some simple heuristics like merging noun token sequences together to extract ngrams but they are too simplistic and very error prone.

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#9

Post by echelon » Mon, Jan 13, 2020, 6:27 PM UTC

I'm very familiar with the TTS, VC, and other "audio-shaped" spaces, but I've never delved into NLP.

What problems can you solve with NLP? Sentiment analysis? Semantic analysis? Translation?

What cool problems are there?

Re: Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines

#10

Post by mark_l_watson » Mon, Jan 13, 2020, 6:33 PM UTC

I love the work done and made freely available by both spaCy and HuggingFace.

I had my own NLP libraries for about 20 years, simple ones were examples in my books, and more complex and not so understandable ones I sold as products and pulled in lots of consulting work with.

I have completely given up my own work developing NLP tools, and generally I use the Python bindings (via the Hy language (hylang) which is a Lisp that sits on top of Python) for spaCy, huggingface, TensorFlow, and Keras. I am retired now but my personal research is in hybrid symbolic and deep learning AI.