Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advanced NLP with SpaCy

course.spacy.io

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Advanced NLP with SpaCy

#51

Post by squeaky-clean »

yorwba wrote: post #40
croes wrote: post #39

Earlier quoted context omitted.

LLMs is non deterministic, so tomorrow the same prompt can give you a totally different result. That's what renders it nearly useless for such tasks, because you always have to check everything.

There's nothing about LLMs that is inherently nondeterministic. Sure, if you're using some API you have no control over, anything could happen. But if you run it on your own hardware, you can make it as deterministic as any classic NLP approach. And whether you always have to check everything is a separate question from nondeterminism. You could have a deterministic heuristic that is often wrong in a domain where mis…

ChatGPT 4 is inherently non-deterministic. Even across the same exact model version with temperature set to 0.

https://ai.plainenglish.io/gpt-4-is-non-deterministic-af2028...

Re: Advanced NLP with SpaCy

#52

Post by squeaky-clean »

dartos wrote: post #44
croes wrote: post #39

Earlier quoted context omitted.

LLMs is non deterministic, so tomorrow the same prompt can give you a totally different result. That's what renders it nearly useless for such tasks, because you always have to check everything.

Only the training is non deterministic. As long as you use the same seed for inference, it’ll give you the same answer. ChatGPT changes this seed under the hood with each request

Even if you keep the same seed, it is non-deterministic.

https://ai.plainenglish.io/gpt-4-is-non-deterministic-af2028...

Re: Advanced NLP with SpaCy

#53

Post by yorwba »

squeaky-clean wrote: post #51
yorwba wrote: post #40

Earlier quoted context omitted.

There's nothing about LLMs that is inherently nondeterministic. Sure, if you're using some API you have no control over, anything could happen. But if you run it on your own hardware, you can make it as deterministic as any classic NLP approach. And whether you always have to check everything is a separate question from nondeterminism. You could have a deterministic heuristic that is often wrong in a domain where mis…

ChatGPT 4 is inherently non-deterministic. Even across the same exact model version with temperature set to 0. https://ai.plainenglish.io/gpt-4-is-non-deterministic-af2028...

That's what I meant by "if you're using some API you have no control over, anything could happen." OpenAI could offer access to a fully deterministic version at a higher price point, but they choose not to and there's nothing you can do about that.

FYI, your source is medium-walled blog spam of https://152334h.github.io/blog/non-determinism-in-gpt-4/

Re: Advanced NLP with SpaCy

#54

Post by ianbicking »

syllogism wrote: post #46
ianbicking wrote: post #43

Earlier quoted context omitted.

"If you know what those types of article are you can separate them out and make sure they're represented in the prompt. But now we're back at needing a tonne of domain knowledge about your problem -- that's like having to write custom features to make your model work" I think this is where the different perspectives come into play. If you're an NLP practitioner you are thinking, oh no! I need to know a lot about audi…

I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML. I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it s…

"For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task."

I think this is a good example of how decomposed tasks can feel very different from goal-oriented task definitions.

We can imagine a goal like "I want to know when one of my competitors shows up in the news." Now you have a bunch of tasks: entity extraction, determining the subject of an article, maybe categorizing articles. Then you can define a pipeline and conditional to trigger a notification. And you might get great accuracy on each of these.

But the goal is really about getting actionable information. In practice the approach above creates a ton of alerts, and the person receiving them will filter through them, ignore a bunch, have to figure out what is really new, etc. An LLM could do things like accumulate a running set of background knowledge, identifying what information is truly "new" (and in a granular way, not just detecting duplicate articles). You can tell the LLM all kinds of details about what you are interested in, "categories" that are completely inaccessible to traditional NLP because they are described by higher-level concepts or have to be combined with history or user-provided context (something that happens naturally in a prompt).

Traditional NLP feels very industrial to me. Factories can be very productive and high volume, but they redefine the tasks to satisfy their processes. Individuals don't interface with factories, and factories don't serve individuals.

Re: Advanced NLP with SpaCy

#55

Post by squeaky-clean »

yorwba wrote: post #53
squeaky-clean wrote: post #51

Earlier quoted context omitted.

ChatGPT 4 is inherently non-deterministic. Even across the same exact model version with temperature set to 0. https://ai.plainenglish.io/gpt-4-is-non-deterministic-af2028...

That's what I meant by "if you're using some API you have no control over, anything could happen." OpenAI could offer access to a fully deterministic version at a higher price point, but they choose not to and there's nothing you can do about that. FYI, your source is medium-walled blog spam of https://152334h.github.io/blog/non-determinism-in-gpt-4/

If you'd read the source you linked you'd see that no, the system is inherently non-deterministic. They cannot sell you a deterministic version of GPT-4. There is no such thing. They could sell you a deterministic version of some other inferior model.

Re: Advanced NLP with SpaCy

#56

Post by yorwba »

squeaky-clean wrote: post #55
yorwba wrote: post #53

Earlier quoted context omitted.

That's what I meant by "if you're using some API you have no control over, anything could happen." OpenAI could offer access to a fully deterministic version at a higher price point, but they choose not to and there's nothing you can do about that. FYI, your source is medium-walled blog spam of https://152334h.github.io/blog/non-determinism-in-gpt-4/

If you'd read the source you linked you'd see that no, the system is inherently non-deterministic. They cannot sell you a deterministic version of GPT-4. There is no such thing. They could sell you a deterministic version of some other inferior model.

From the source, which I read:

As a consequence, the model is no longer deterministic at the sequence-level, but only at the batch-level

They could sell you a version where you shove in a whole batch, pay $batch_size as much, and always get the same result for the same batch.

Re: Advanced NLP with SpaCy

#57

Post by syllogism »

famouswaffles wrote: post #47
syllogism wrote: post #19

Earlier quoted context omitted.

(Original author of spaCy and Explosion CTO here) Okay so, first some terminology. LLMs can mean a bunch of different things, people call models the size of BERT LLMs sometimes. So let's talk specifically about in-context learning (ICL) with either zero or a few examples. So we'll say LLM ICL, and contrast that with techniques where you annotate enough data to train with, which might only be something like 10-40 hour…

>ICL is generally much worse than training a model in accuracy, and you generally don't need much training data to surpass ICL in accuracy. For the same model is a huge asterisk you seem to be missing here. Finetuned GPT-4 is better than ICL Gpt-4 and so on but there's no guarantee that finetuned GPT-3 will be better than ICL GPT-4 like how 4 beats the finetuned Med-Palm on medical domain tests. You can train bespoke…

I'm making the stronger claim that fine-tuned BERT is almost always better than ICL GPT4. That is, you beat ICL GPT4 with a model of less than 1b parameters. These are the results in the literature.

Fine-tuning GPT3 is actually a strange middle-ground: you're constrained to solve the tasks via prompting, rather than attaching output layers and doing gradient descent over a loss function that directly matches the task. So fine-tuning LLMs for these classification tasks is indeed often worse than just using the larger model.

The other thing is, as you scale up the model the ratio of output weights to the whole model weights gets quite out of balance. This introduces some technical challenges, and so it's kind of tough to make it all work with medium size LLMs of around 7b parameters or so. In contrast the process of fine-tuning BERT-sized stuff is really well explored, and it's easy to find more or less push-button solutions.

Re: Advanced NLP with SpaCy

#58

Post by syllogism »

ianbicking wrote: post #54
syllogism wrote: post #46

Earlier quoted context omitted.

I know what you mean about the domain knowledge, and it's a thing that's a bit different from the previous situation with the feature engineering. The problem with feature engineering for linear models was you really had to understand the domain _and_ the ML. I do think there's a similarity in how creative you need to be though. It means that applying LLMs to new problems isn't as straight-forward as people make it s…

"For instance, let's say you want to trigger some alert when a particular kind of article is published or a particular kind of message is sent. Triggering the alert is a boolean thing, so that has to be a classification task." I think this is a good example of how decomposed tasks can feel very different from goal-oriented task definitions. We can imagine a goal like "I want to know when one of my competitors shows u…

I think the 'industrial' or assembly-line analogy is probably good, and I see what you mean about the alternative system design. Thanks for explaining the other approach.

I'll put it this way. If you want to integrate ML into a product, or even a system with lots of internal users, you end up increasingly towards the 'factory' approach.

Re: Advanced NLP with SpaCy

#59

Post by dartos »

squeaky-clean wrote: post #52
dartos wrote: post #44

Earlier quoted context omitted.

Only the training is non deterministic. As long as you use the same seed for inference, it’ll give you the same answer. ChatGPT changes this seed under the hood with each request

Even if you keep the same seed, it is non-deterministic. https://ai.plainenglish.io/gpt-4-is-non-deterministic-af2028...

That link is dead. Either way, you can’t interact with gpt4 without openai’s api, so you don’t know what they’re doing in between your request and inference.

But LLMs are deterministic when all the parameters are the same between inferences

Re: Advanced NLP with SpaCy

#60

Post by fogx »

bob1029 wrote: post #35
tomohelix wrote: post #3

No offense, but isn't the NLP field effectively solved with the creation of LLMs, or at least for the majority of the tasks you would expect from an NLP application? I am sure you can find some special areas or niches where traditional NLP approaches would outcompete a black box like LLMs. But with the LLMs becoming much more efficient now after quantization to the point you can run them locally, I think there is a g…

The #1 thing for me right now is determinism and traceability. I am in a "serious" business domain and non-determinism is a big no-no. We have to be able to justify everything in a traditional sense at the end of the day. Explaining to a regulator that we declined a customer as a consequence of a vast, unthinkable sea of weights and biases is not going to fly. For each predicted output token, I want to know exactly w…

> For each predicted output token, I want to know exactly which source document(s) were utilized including indices from those documents and relevant statistics.

You don't do this for any other kind of decision or tool, why do you need it for LLMs?

i think in truth you need a source that convinces you (or the regulator) that your choice is acceptable, so that you can pass off the responsibility. If the LLM were to give you an answer backed by relevant (cited) source documents of regulations and a good explanation it would make no difference compared to a human worker doing the same. -> this is already possible

Post reply on HN