Advanced NLP with spaCy v3 • Hacker News Board

Re: Advanced NLP with spaCy v3

#11

Post by hooande » Fri, Dec 10, 2021, 6:18 PM UTC

new_stranger wrote: post #9

minimaxir wrote: post #4

A relatively underdiscussed quirk of the rise of superlarge language models like GPT-3 for certain NLP tasks is that since those models have incorporated so much real world grammar, there's no need to do advanced preprocessing and can just YOLO and work with generated embeddings instead without going into spaCy's (excellent) parsing/NER features. OpenAI recently released an Embeddings API for GPT-3 with good demos an…

I imagine it being very useful to understand what you just said

lol. a rough translation is that the new super language models are good enough that you don't have to keep track of specific parts of speech in your programming. if you look at the arrays of floating point weights that underlie gpt-3 etc, you can use them to match present participle phrases with other present participle phrases and so forth

this is of course a correct and prescient observation. minimaxir is kind of an NLP final boss, so I wouldn't expect most people to be able to follow everything he says

Re: Advanced NLP with spaCy v3

#12

Post by Ldorigo » Fri, Dec 10, 2021, 6:28 PM UTC

master_yoda_1 wrote: post #10

Quoted post unavailable.

Ah, yes. The tried-and-true method of "just selling the hype" with an open source library that everyone can use for free.

Re: Advanced NLP with spaCy v3

#13

Post by minimaxir » Fri, Dec 10, 2021, 6:31 PM UTC

hooande wrote: post #11

new_stranger wrote: post #9

Earlier quoted context omitted.

I imagine it being very useful to understand what you just said

lol. a rough translation is that the new super language models are good enough that you don't have to keep track of specific parts of speech in your programming. if you look at the arrays of floating point weights that underlie gpt-3 etc, you can use them to match present participle phrases with other present participle phrases and so forth this is of course a correct and prescient observation. minimaxir is kind of a…

I don't think it's more of a final boss thing: IMO working with embeddings/word vectors is easier, even in the basest case such as word2vec/GloVe, to understand than some of the more conventional NLP techniques (e.g. bag of words/TF-IDF).

The spaCy tutorials in the submission also have a section on word vectors.

Re: Advanced NLP with spaCy v3

#14

Post by Xenoamorphous » Fri, Dec 10, 2021, 6:35 PM UTC

artembugara wrote: post #2

We've been using spaCy a lot for the past few months. Mostly for non-production use cases, however, I can say that it is the most robust framework for NLP at the moment. V3 added support for transformers: that's a killer feature as many models from https://huggingface.co/docs/transformers/index work great out of the box. At the same time, I found NER models provided by spaCy to have a low accuracy while working with…

I don’t know how it compares with other paid alternatives (like Google’s or Amazon’s) but spaCy’s NER was pretty close to the (paid) service we were using (IBM) to the point we ditched IBM. Also for news articles.

But yeah disambiguation/entity linking would be nice.

Re: Advanced NLP with spaCy v3

#15

Post by artembugara » Fri, Dec 10, 2021, 6:44 PM UTC

Xenoamorphous wrote: post #14

artembugara wrote: post #2

We've been using spaCy a lot for the past few months. Mostly for non-production use cases, however, I can say that it is the most robust framework for NLP at the moment. V3 added support for transformers: that's a killer feature as many models from https://huggingface.co/docs/transformers/index work great out of the box. At the same time, I found NER models provided by spaCy to have a low accuracy while working with…

I don’t know how it compares with other paid alternatives (like Google’s or Amazon’s) but spaCy’s NER was pretty close to the (paid) service we were using (IBM) to the point we ditched IBM. Also for news articles. But yeah disambiguation/entity linking would be nice.

I'd be happy to chat more if you want.

Re: Advanced NLP with spaCy v3

#16

Post by dang » Fri, Dec 10, 2021, 7:01 PM UTC

master_yoda_1 wrote: post #10

Quoted post unavailable.

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

Re: Advanced NLP with spaCy v3

#17

Post by mtqwerty » Fri, Dec 10, 2021, 7:05 PM UTC

minimaxir wrote: post #4

A relatively underdiscussed quirk of the rise of superlarge language models like GPT-3 for certain NLP tasks is that since those models have incorporated so much real world grammar, there's no need to do advanced preprocessing and can just YOLO and work with generated embeddings instead without going into spaCy's (excellent) parsing/NER features. OpenAI recently released an Embeddings API for GPT-3 with good demos an…

Readjusting expectations for pre-processing was one of the biggest differences I noticed going from NLP courses to working on NLP in production. For the amount of pre-processing learning material there is, I expected it to be much more important in practice.

I feel lucky to gotten into NLP when I did (learning in 2017/2018 and working in the beginning of 2020). Changing our system from glove to BERT was super exciting and a great way to learn about the drawbacks and benefits of each.

Re: Advanced NLP with spaCy v3

#18

Post by coding123 » Fri, Dec 10, 2021, 7:13 PM UTC

master_yoda_1 wrote: post #10

Quoted post unavailable.

That's a huge part of software development. Wrapping things to be more concise, use-case driven. I mean most software developers are just placing a veneer over something more complex. That's pretty much all we do.

Re: Advanced NLP with spaCy v3

#19

Post by brd » Fri, Dec 10, 2021, 7:45 PM UTC

artembugara wrote: post #2

We've been using spaCy a lot for the past few months. Mostly for non-production use cases, however, I can say that it is the most robust framework for NLP at the moment. V3 added support for transformers: that's a killer feature as many models from https://huggingface.co/docs/transformers/index work great out of the box. At the same time, I found NER models provided by spaCy to have a low accuracy while working with…

I really appreciate how accessible SpaCy has made NLP work but their NER is definitely low accuracy.

Where stem/lem felt critical to successful NLP processing a few years ago, we've found stem/lem work to be much less important for downstream tasks when transformer based models are involved.

For topic extraction stem/lem still seems to do a lot to improve accuracy and for rules based approaches I can still see how it would facilitate more efficient processing at scale. I'd be curious to hear your experience fine tuning and/or training new models after stem/lem processing with transformers, we've admittedly done little testing to see how transformers actually performer if properly tuned to post-processed data.

Re: Advanced NLP with spaCy v3

#20

Post by artembugara » Fri, Dec 10, 2021, 7:48 PM UTC

brd wrote: post #19

artembugara wrote: post #2

We've been using spaCy a lot for the past few months. Mostly for non-production use cases, however, I can say that it is the most robust framework for NLP at the moment. V3 added support for transformers: that's a killer feature as many models from https://huggingface.co/docs/transformers/index work great out of the box. At the same time, I found NER models provided by spaCy to have a low accuracy while working with…

I really appreciate how accessible SpaCy has made NLP work but their NER is definitely low accuracy. Where stem/lem felt critical to successful NLP processing a few years ago, we've found stem/lem work to be much less important for downstream tasks when transformer based models are involved. For topic extraction stem/lem still seems to do a lot to improve accuracy and for rules based approaches I can still see how it…

Did you try something like autoNLP by huggingface?