Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advanced NLP with SpaCy

course.spacy.io

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Advanced NLP with SpaCy

#41

Post by abhgh »

tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

"NLP is basically solved" --- somewhat but not the entirely yet. I work on a variety of specialized NLP use-cases for clients and there are different strengths these approaches have. The big thing with LLMs is the ability to deploy something quickly if (a) if you can craft an appropriate prompt, (b) put up enough guardrails to stop surfacing hallucinated responses to users. For assistant or co-pilot kind of systems (b) is somewhat easy to deal with, since the user is expected to curate or edit the LLM's responses, hence their proliferation in such use-cases. Note though: if the LLM sounds authoritative enough it might bias the user into believing it is right - not a big problem when the user is relying on it for generating good prose, but this is a problem when presenting facts (esp. based on a specialized knowledge base).

The downsides are (a) latency, (b) cost, and (c) the need for specialized training. In some applications, you require near real-time responses, and smaller models are still better here. The cost angle is a little tricky - it depends both on the volume of calls you want to make and if this cost translates into revenue for the possibly incremental benefit you derive from an LLM. As an example, lets say you have a chatbot that does NER/slot-filling using a spacy or stanza today, and lets also assume ChatGPT can do better - does the incremental accuracy, that comes at a cost since you're paying OpenAI - translate into incremental revenue (or profit)? I am not sure what the answer here is - its probably a NO right now, but there is a positive deferred benefit - as your chatbot solution becomes better in many small and large ways, it might sell better in the future. The specialized training part is when you can gather a use-case-specific dataset that can get a fairly good accuracy (comparable or greater than an LLM), esp. considering (a) and (b). Note that these concerns are largely true even for self-hosted LLMs like llama - just that the precise breakeven point changes.

Re: Advanced NLP with SpaCy

#42

Post by billythemaniam »

rhdunn wrote: post #16
tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

LLMs are useful for things like predicting/generating text, and summarizing text. They are not useful if you want to do other NLP tasks that include things like: 1. Identifying (and highlighting/extracting) the language that spans of text are in within a different language (e.g. a French phrase in English). 2. Text search and highlighting, where you need to do things like performing word stemming or lemmatization on…

I agree LLMs alone aren't good at search but their embeddings replace the need for stemming, manual synonym lists, etc in most cases. LLMs can also be used for query understanding which can improve the keywords submitted to the engine and extracting the best snippet for a highlight. LLMs + search are better than either alone. However LLMs still have an inference performance/cost issue which may make them unsuitable for some search use cases.

Re: Advanced NLP with SpaCy

#43

Post by ianbicking »

syllogism wrote: post #19
tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

(Original author of spaCy and Explosion CTO here) Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hour…

"If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work"

I think this is where the different perspectives come into play.

If you're an NLP practitioner you are thinking, oh no! I need to know a lot about audience intention and how the articles are represented navigationally and the kind of variety people are looking for and how articles might fit multiple categories, etc, etc. And you have to think about these things on a meta level ("prompt engineering"), because you have to instruct the model on how to act in an abstract way.

If you're someone who wants to run a news site then you already are thinking a ton about these things, and probably have a dozen things you'd like to change and adjust, new ways of presenting content, etc. You _wish_ you could be thinking about these domain-specific topics.

What feels like a bug to the NLP practitioner – needing a deep understanding of the domain – is a feature to... everyone else. It's a feature to the people who care most about the results.

The other big perspective difference here, I believe, is how you think about goals. How many tasks ARE categorization? My intuition is that it's a quite small number. There are many tasks that can be implemented with one step as categorization, but that is seldom the task. To the NLP practitioner categorization might seem very prominent – that's when someone calls you up or hands over the work. But with an LLM you might be able to do a much larger portion of the real task, with or without categorization.

Even with a categorization task, when I'm working with an LLM I usually don't produce just a "category", but produce other information at the same time, often using natural language as a first-class data type because it can be fed back into an LLM. In my experience the results are often (usually?) much higher quality because I'm not breaking things down into steps where the inaccuracies propagate between steps, but instead going right for the result, and using a model that can basically "check" itself against general knowledge, scrubbing out nonsensical results during inference. (As a result the remaining inaccuracies often appear plausible and are labeled "hallucinations"... it can make things more challenging, but what we don't see are the multitude of obvious inaccuracies that a more traditional NLP system would create, and which in a sense exist momentarily during the LLM inference.)

Re: Advanced NLP with SpaCy

#44

Post by dartos »

croes wrote: post #39
yorwba wrote: post #20

Earlier quoted context omitted.

1. Prompt: """ The following English text contains several French phrases. List all of them. Text: Gabonese President Ali Bongo Odimba was deposed in a coup d'etat spearheaded by his father's former aide-de-camp Brice Oligui Nguema, shortly after the announcement that Bongo had won the 2023 election. List of French phrases from the text: - """ Response: """ coup d'etat - aide-de-camp """ 2. Prompt: """ Turn all words…

LLMs is non deterministic, so tomorrow the same prompt can give you a totally different result. That's what renders it nearly useless for such tasks, because you always have to check everything.

Only the training is non deterministic. As long as you use the same seed for inference, it’ll give you the same answer.

ChatGPT changes this seed under the hood with each request

Re: Advanced NLP with SpaCy

#45

Post by huksley »

Anyone actually using SpaCy as part of the production API / queue processing? Like 100 RPS or more? I found it to be unstable to serve API requests, unless you dedicate unreasonably extremely overpowered VM instance to it.

How people solve sizing issues for python based APIs with multiple simultaneous requests processing?

Re: Advanced NLP with SpaCy

#46

Post by syllogism »

ianbicking wrote: post #43
syllogism wrote: post #19

Earlier quoted context omitted.

(Original author of spaCy and Explosion CTO here) Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hour…

"If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work" I think this is where the different perspectives come into play. If you're an NLP practitioner you are thinking, oh no! I need to know a lot about audi…

I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML.

I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it seem at first, and isn't necessarily reliable. In contrast, labelling data is something that has a much smoother effort to reward curve for most problems. The experience of labelling the data, training a model and getting it hosted isn't as seamless as it could be -- we're working on that.

I do think classification is pretty fundamental though. The way I see it is, model outputs can be either human-facing, machine facing, or both. If you're going to feed the output into another system, that system wants the data to obey some limited schema, so that you can run logic based on it. For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task. You might want to also attach text in the alert, so that's a human-facing part.

I agree that there's lots of ways that LLMs can be used iteratively, allowing more trade-off of computational cost for accuracy. I just think in a lot of cases, the best way to exploit that is to trade towards as much accuracy as you can get, and use that to create training samples. You can then manually correct the training samples as well --- if they're mostly correct, reviewing them is pretty quick. You can then train and evaluate a smaller model.

Re: Advanced NLP with SpaCy

#47

Post by famouswaffles »

syllogism wrote: post #19
tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

(Original author of spaCy and Explosion CTO here) Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hour…

>ICL is generally much worse than training a model in accuracy, and you generally don't need much training data to surpass ICL in accuracy.

For the same model is a huge asterisk you seem to be missing here. Finetuned GPT-4 is better than ICL Gpt-4 and so on but there's no guarantee that finetuned GPT-3 will be better than ICL GPT-4 like how 4 beats the finetuned Med-Palm on medical domain tests.

You can train bespoke models with worse accuracy.

Re: Advanced NLP with SpaCy

#48

Post by Rant423 »

tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

Sometimes you want deterministic, rule-based results (e.g. https://spacy.io/usage/rule-based-matching) and not a "fuzzy" one (LLMs)

Re: Advanced NLP with SpaCy

#49

Post by [deleted] »

markisus wrote: post #28
gattilorenz wrote: post #9

Earlier quoted context omitted.

I don’t understand what it means to be “solved”. It’s like saying that “architecture is now solved”, “physics is solved”, or “programming is solved”. It’s a field of science and/or engineering, it’s not like we will ever run out of things to try/build/investigate. LLMs work… to a certain extent, with limitations and tradeoffs, and for some things. Would you spend days, money and Co2 to split a huge text corpus in sen…

I take it to mean that there is an effective generally accepted solution or methodology for problems in the field. Bridge building has been largely solved by methods of mathematical and computational structural analysis, manufacturing, and government regulation. We know how to build a bridge. Before the solution was known, designers would just go by intuition and we wouldn’t have any actual assurance that the bridge…

[deleted]

Re: Advanced NLP with SpaCy

#50

Post by gattilorenz »

markisus wrote: post #28
gattilorenz wrote: post #9

Earlier quoted context omitted.

I don’t understand what it means to be “solved”. It’s like saying that “architecture is now solved”, “physics is solved”, or “programming is solved”. It’s a field of science and/or engineering, it’s not like we will ever run out of things to try/build/investigate. LLMs work… to a certain extent, with limitations and tradeoffs, and for some things. Would you spend days, money and Co2 to split a huge text corpus in sen…

I take it to mean that there is an effective generally accepted solution or methodology for problems in the field. Bridge building has been largely solved by methods of mathematical and computational structural analysis, manufacturing, and government regulation. We know how to build a bridge. Before the solution was known, designers would just go by intuition and we wouldn’t have any actual assurance that the bridge…

I think NLP is closer to architecture than to bridge building (and even there, I’m sure we’re still probably researching stuff to know more about how to calculate the stresses and whatnot)
Post reply on HN