Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Has LLM killed traditional NLP?

medium.com

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Has LLM killed traditional NLP?

#71

Post by simonw »

axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

What NLP approaches are you using to solve the "is '20 bottles of ferric chloride' a service or a product?" problem?

Re: Has LLM killed traditional NLP?

#73

Post by simonw »

alexwebb2 wrote: post #23
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

I think your intuition on this might be lagging a fair bit behind the current state of LLMs. System message: answer with just "service" or "product" User message (variable): 20 bottles of ferric chloride Response: product Model: OpenAI GPT-4o-mini $0.075/1Mt batch input * 27 input tokens * 10M jobs = $20.25 $0.300/1Mt batch output * 1 output token * 10M jobs = $3.00 It's a sub-$25 job. You'd need to be doing 20 times…

You might be able to use an even cheaper model. Google Gemini 1.5 Flash 8B is Input: $0.04 / Output: $0.15 per 1M tokens.

17 input tokens and 2 output tokens * 10 million jobs = 170,000,000 input tokens, 20,000,000 output tokens... which costs a total of $6.38 https://tools.simonwillison.net/llm-prices

As for rate limits, https://ai.google.dev/pricing#1_5flash-8B says 4,000 requests per minute and 4 million tokens per minute - so you could run those 10 million jobs in about 2500 minutes or 42 hours. I imagine you could pull a trick like sending 10 items in a single prompt to help speed that up, but you'd have to test carefully to check the accuracy effects of doing that.

Re: Has LLM killed traditional NLP?

#74

Post by simonw »

hnfong wrote: post #70
FloorEgg wrote: post #68

Earlier quoted context omitted.

Run them all in parallel with a cloud function in less than a minute?

Obviously all the LLM API providers have a rate limit. Not a fan of GP's sarcastic tone, but I suppose many of us would like to know roughly what that limit would be for a small business using such APIs.

Surprisingly, DeepSeek doesn't have a rate limit: https://api-docs.deepseek.com/quick_start/rate_limit

I've heard from people running 100+ prompts in parallel against it.

Re: Has LLM killed traditional NLP?

#75

Post by mikeocool »

segmondy wrote: post #65
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250…

If you have to classify user input as they’re inputting it to provide a response — so it can’t be batched - 2 seconds could potentially be really slow.

Though LLMs sure have made creating training data to train old school models for those cases a lot easier.

Re: Has LLM killed traditional NLP?

#76

Post by itissid »

LLM Design/Use has only about as much to with engineering as building a plane has to do with actually flying it.

Every business is kind of a unicorn in its problems NLP is a small part of it. Like even if it did perform cheaply enough to do NLP, how would you replace parts like: 1. Evaluation system that uses Calibration(Human labels) 2. Ground Truth Collection(Human + sometimes semi automated) 3. QA testing by end users.

Even if LLMs made it easier to do NLP there are correlations with the above which means your NLP process is hugely influenced so much that you still need an engineer. If you have an engineer who only for doing NLP and nothing else you are quite hyper specialized like to the extent you are only building planes 0.01%: of the engineering work out there.

Re: Has LLM killed traditional NLP?

#77

Post by vunderba »

blindriver wrote: post #29
dartos wrote: post #26

Earlier quoted context omitted.

> LLM costs Inference costs, not training costs. > The fact that you can replace programmers You can’t… not for any real project. For quick mockups they’re serviceable > That’s sort of like asking a horse and buggy driver whether automobiles Kind of an insult to OP, no? Horse and buggy drivers were not highly educated experts in their field. Maybe take the word of domain experts rather than AI company marketing teams…

> Maybe take the word of domain experts rather than AI company marketing teams. Appeal to authority is a well known logical fallacy. I know how dead NLP is personally because I’ve never been able to get NLP working but once ChatGPT came around, I was able to classify texts extremely easily. It’s transformational. I was able to get ChatGPT to classify posts based on how political it was from a scale of 1 to 10 and whi…

> NLPs are dead in the water right now, except in terms of cost.

False.

With all due respect, the fact that you're referring to natural language parsing as "NLPs" makes me question whether you have any experience or modest knowledge around this topic, so it's rather bold of you to make such sweeping generalizations.

It works for your use case because you're just one person running it on your home computer with consumer hardware. Some of us have to run NLP related processing (POS taggers, keyword extraction, etc) in a professional environment at tremendous scale, and reaching for an LLM would absolutely kill our performance.

Re: Has LLM killed traditional NLP?

#78

Post by w10-1 »

alexwebb2 wrote: post #23
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

I think your intuition on this might be lagging a fair bit behind the current state of LLMs. System message: answer with just "service" or "product" User message (variable): 20 bottles of ferric chloride Response: product Model: OpenAI GPT-4o-mini $0.075/1Mt batch input * 27 input tokens * 10M jobs = $20.25 $0.300/1Mt batch output * 1 output token * 10M jobs = $3.00 It's a sub-$25 job. You'd need to be doing 20 times…

The question is not average cost but marginal cost of quality - same as voice recognition, which had relatively low uptake even at ~2-4% error rates due to context switching costs for error correction.

So you'd have to account for the work of catching the residue of 2-8%+ error from LLMs. I believe the premise is for NLP, that's just incremental work, but for LLM's that could be impossible to correct (i.e., cost per next-percentage-correction explodes), for lack of easily controllable (or even understandable) models.

But it's most rational in business to focus on the easy majority with lower costs, and ignore hard parts that don't lead to dramatically larger TAM.

Re: Has LLM killed traditional NLP?

#79

Post by jdietrich »

hnfong wrote: post #70
FloorEgg wrote: post #68

Earlier quoted context omitted.

Run them all in parallel with a cloud function in less than a minute?

Obviously all the LLM API providers have a rate limit. Not a fan of GP's sarcastic tone, but I suppose many of us would like to know roughly what that limit would be for a small business using such APIs.

The rate limits for Gemini 1.5 Flash are 2000 requests per minute and 4 million tokens per minute. Higher limits are available on request.

https://ai.google.dev/pricing#1_5flash

4o-mini's rate limits scale based on your account history, from 500RPM/200,000TPM to 30,000RPM/150,000,000TPM.

https://platform.openai.com/docs/guides/rate-limits

Re: Has LLM killed traditional NLP?

#80

Post by WildGreenLeave »

segmondy wrote: post #65
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250…

Correct me if I'm wrong, but, if you run multiple inferences at the same time on the same GPU you will need load multiple models in the vram and the models will fight for resources right? So running 10 parallel inferences will slow everything down 5 times right? Or am I missing something?
Post reply on HN