Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Has LLM killed traditional NLP?

medium.com

Post reply on HN
Print view View original

81–90 of 159 posts

Re: Has LLM killed traditional NLP?

#81

Post by aeternum »

WildGreenLeave wrote: post #80
segmondy wrote: post #65

Earlier quoted context omitted.

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250…

Correct me if I'm wrong, but, if you run multiple inferences at the same time on the same GPU you will need load multiple models in the vram and the models will fight for resources right? So running 10 parallel inferences will slow everything down 5 times right? Or am I missing something?

No, the key is to use the full context window so you structure the prompt as something like: For each line below, repeat the line, add a comma then output whether it most closely represents a product or service:

20 bottles of ferric chloride

salesforce

...

Re: Has LLM killed traditional NLP?

#82

Post by rlt »

FloorEgg wrote: post #68
axegon_ wrote: post #61

Earlier quoted context omitted.

Yeah... Let's talk time needed for 10M prompts and how that fits into a daily pipeline. Enlighten us, please.

Run them all in parallel with a cloud function in less than a minute?

Also can’t you just combine multiple classification requests into a single prompt?

Re: Has LLM killed traditional NLP?

#83

Post by minimaxir »

Kuinox wrote: post #10
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

Prompt caching would lower the cost, later similar tech would lower the inference cost too. You have less than 25 tokens, thats between 1-5$. There may be some use case but I'm not convinced with the one you gave.

So there's a bit of an issue with prompt caching implementations: for both OpenAI API and Claude's API, you need a minimum of 1024 tokens to build the cache for whatever reason. For simple problems, that can be hard to hit and may require padding the system prompt a bit.

Re: Has LLM killed traditional NLP?

#84

Post by LeafItAlone »

scarface_74 wrote: post #14

For my use case, definitely. I have worked on AWS Connect (online call center) and Amazon Lex (the backing NLP engine) projects. Before LLMs, it was a tedious process of trying to figure out all of the different “utterances” that people could say and the various languages you had to support. With LLMs, it’s just prompting https://chatgpt.com/share/678bab08-f3a0-8010-82e0-32cff9c0b4... I used something like this using…

That’s a great example and I understand it was intentionally simple but highlighted how LLMs need care with use. Not that this example is very related to NLP:

My prompt: `>`

The response: ``` { "origin": ["PDX"], "destination": ["HAV"], "date": "2025-04-01", "departure_time": null, "preferences": null } ```

Of course I meant Portland Maine (PWM), there is more than one airport option in Cuba than HAV, and it got the date wrong, since Easter is April 20 this year.

Re: Has LLM killed traditional NLP?

#85

Post by tossandthrow »

leobg wrote: post #49

There are AI bros that will call an LLM to do what you could do with a regex. I’ve seen people do the chunking for RAG using an LLM…

If you think about chunking as "take x characters" then using LLMs is a poor idea.

But syntactic chunking also works really poorly for any serious application as you loose basically all context.

Semantic chunking, however, is a task you absolutely would use LLMs for.

Re: Has LLM killed traditional NLP?

#86

Post by thangalin »

I created an NLP library to help curl straight quotes into curly quotes. Last I checked, LLMs struggled to curl the following straight quotation marks:

    ''E's got a 'ittle box 'n a big 'un,' she said, 'wit' th' 'ittle 'un 'bout 2'×6". An' no, y'ain't cryin' on th' "soap box" to me no mo, y'hear. 'Cause it 'tweren't ever a spec o' fun!' I says to my frien'.
The library is integrated into my Markdown editor, KeenWrite (https://keenwrite.com/), to correctly curl quotation marks into entities before passing them over to ConTeXt for typesetting. While there are other ways to indicate opening and closing quotation marks, none are as natural to type in plain text as straight quotes. I would not trust an LLM curl quotation marks accurately.

For the curious, you can try it at:

https://whitemagicsoftware.com/keenquotes/

If you find any edge cases that don't work, do let me know. The library correctly curls my entire novel. There are a few edge cases that are completely ambiguous, however, that require semantic knowledge (part-of-speech tagging), which I haven't added. PoS tagging would be a heavy operation that could prevent real-time quote curling for little practical gain.

The lexer, parser, and test cases are all open source.

https://gitlab.com/DaveJarvis/KeenQuotes/-/tree/main/src/mai...

Re: Has LLM killed traditional NLP?

#87

Post by axegon_ »

segmondy wrote: post #65
axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250…

FFS... "Lots of writers, few readers". Read again and do the math: 2 seconds, multiply that by 10 million records which contain this, as well as "alarm installation in two locations" and a whole bunch of other crap with little to no repetition (<2%) and where does that get you? 2 * 10,000,000 = 20,000,000 SECONDS!!!! A day has 86,400 seconds (24 * 3600 = 86,400). The data pipeline needs to finish in <24 hours. Everyone needs to get this into their heads somehow: LLM's are not a silver bullet. They will not cure cancer anytime soon, nor will they be effective or cheap enough to run at massive scale. And I don't mean cheap as in "oh, just get openai subscription hurr durr". Throwing money mindlessly into something is never an effective way to solve a problem.

Re: Has LLM killed traditional NLP?

#88

Post by derbaum »

One of the things I'm still struggling with when using LLMs over NLP is classification against a large corpus of data. If I get a new text and I want to find the most similar text out of a million others, semantically speaking, how would I do this with an LLM? Apart from choosing certain pre-defined categories (such as "friendly", "political", ...) and then letting the LLM rate each text on each category, I can't see a simple solution yet except using embeddings (which I think could just be done using BERT and does not count as LLM usage?).

Re: Has LLM killed traditional NLP?

#89

Post by crystal_revenge »

axegon_ wrote: post #6

No, it has not and will not in the foreseeable future. This is one of my responsibilities at work. LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost. LLMs are great at mid-level complexity tasks given a reasonable volume of data - they can take away the tedious job of figuring out what you are looking at or even come up with some basic…

> LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost.

What? That's simply not true.

Current embedding models are incredibly fast and cheap and will, in the vast majority of NLP tasks, get you far better results than any local set of features you can develop yourself.

I've also done this at work numerous times, and have been working on various NLP tasks for over a decade now. For all future traditional NLP tasks the first pass is going to be to get fetch LLM embeddings and stick on a fairly simple classification model.

> One prompt? Fair. 10? Still ok. 100? You're pushing it. 10M - get help.

"Prompting" is not how you use LLMs for classification tasks. Sure you can build 0-shot classifiers for some tricky tasks, but if you're doing classification for documents today and you're not starting with an embedding model you're missing some easy gains.

Re: Has LLM killed traditional NLP?

#90

Post by thaumasiotes »

derbaum wrote: post #88

One of the things I'm still struggling with when using LLMs over NLP is classification against a large corpus of data. If I get a new text and I want to find the most similar text out of a million others, semantically speaking, how would I do this with an LLM? Apart from choosing certain pre-defined categories (such as "friendly", "political", ...) and then letting the LLM rate each text on each category, I can't see…

Take two documents.

Feed one through an LLM, one word at a time, and keep track of words that experience greatly inflated probabilities of occurrence, compared to baseline English. "For" is probably going to maintain a level of likelihood close to baseline. "Engine" is not.

Do the same thing for the other one.

See how much overlap you get.

Post reply on HN