Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

github.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#61

Post by leobg »

coder68 wrote: post #14
roadside_picnic wrote: post #7

A friend, who also has a background in NLP, was asking me the other day "Is there still even a need for traditional NLP in the age of LLMs?" This is one of the under-discussed areas of LLMs imho. For anything that would have have required either word2vec embeddings of a tf-idf representation (classification tasks, sentiment analysis, etc) there are rare exceptions where it wouldn't just be better to start with a sema…

I have been working on text classification tasks at work, and I have found that for my particular use-case, LLMs are not performing well at all. I have spent a few thousand dollars trying, and I have tried everything from few-shot to asking simple binary yes/no questions, and I have had mixed success. I have stopped trying to use LLMs for this project and switched to discriminative models (Logistic Regression with TF…

If I have 1,000 labeled examples for a classification task, I’ll expand that into a training dataset using augmentation, and then finetune a small model like RoBERTa. It’s fast, cheap, accurate — and predictable.

Others have had success with SetFit as the training framework and Ettin as the base model.

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#62

Post by coder68 »

leobg wrote: post #61
coder68 wrote: post #14

Earlier quoted context omitted.

I have been working on text classification tasks at work, and I have found that for my particular use-case, LLMs are not performing well at all. I have spent a few thousand dollars trying, and I have tried everything from few-shot to asking simple binary yes/no questions, and I have had mixed success. I have stopped trying to use LLMs for this project and switched to discriminative models (Logistic Regression with TF…

If I have 1,000 labeled examples for a classification task, I’ll expand that into a training dataset using augmentation, and then finetune a small model like RoBERTa. It’s fast, cheap, accurate — and predictable. Others have had success with SetFit as the training framework and Ettin as the base model.

oh this seems like an interesting idea, what tactics do you use for augmentation? For my own use-case, I think I could reorder semantic chunks, or maybe randomly delete pieces, but curious what tactics you use!

I have also considered training a small language model for synthetic data generation.

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#63

Post by littlekey »

coder68 wrote: post #14
roadside_picnic wrote: post #7

A friend, who also has a background in NLP, was asking me the other day "Is there still even a need for traditional NLP in the age of LLMs?" This is one of the under-discussed areas of LLMs imho. For anything that would have have required either word2vec embeddings of a tf-idf representation (classification tasks, sentiment analysis, etc) there are rare exceptions where it wouldn't just be better to start with a sema…

I have been working on text classification tasks at work, and I have found that for my particular use-case, LLMs are not performing well at all. I have spent a few thousand dollars trying, and I have tried everything from few-shot to asking simple binary yes/no questions, and I have had mixed success. I have stopped trying to use LLMs for this project and switched to discriminative models (Logistic Regression with TF…

Doesn't that mean having to go back to manually labeling examples? That can be a big hurdle compared to just zero-few shotting some stuff into the LLM prompt. Unless there's something I'm misunderstanding about your approach. Or maybe it's possible to do an unsupervised clustering step on the vectors to get the labeled categories that you can then pass to the supervised classification model. Though I guess that would depend on how strictly defined the target categories are for the use case in question.

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#64

Post by coder68 »

littlekey wrote: post #63
coder68 wrote: post #14

Earlier quoted context omitted.

I have been working on text classification tasks at work, and I have found that for my particular use-case, LLMs are not performing well at all. I have spent a few thousand dollars trying, and I have tried everything from few-shot to asking simple binary yes/no questions, and I have had mixed success. I have stopped trying to use LLMs for this project and switched to discriminative models (Logistic Regression with TF…

Doesn't that mean having to go back to manually labeling examples? That can be a big hurdle compared to just zero-few shotting some stuff into the LLM prompt. Unless there's something I'm misunderstanding about your approach. Or maybe it's possible to do an unsupervised clustering step on the vectors to get the labeled categories that you can then pass to the supervised classification model. Though I guess that would…

To some degree manual labeling has to be done anyway, just to validate that any approach works at all, you'll always need ground truth from somewhere. What I suggested is that zero/few-shotting might not be good enough, depending on the problem. Labeling ~1000 samples isn't too bad, I've done it by hand a few times now. If you can source a high quality positive signal from somewhere (e.g. user-behavioral data), even better.

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#65

Post by coder68 »

IanCal wrote: post #55
coder68 wrote: post #46

Earlier quoted context omitted.

I think the idea of backoff by ratcheting up complexity here is a very good idea, thanks for your suggestions.

Happy to help - this is a thing I’ve employed multiple times for real cases. One big benefit is that it uses the cheapest and most understandable approaches for the majority of cases, and scales up quite nicely. It has a neat place for very custom issues to be fixed too. There will always be some things that simple approaches think are clear but aren’t, which is awkward but then all pipelines end up with that somewhe…

To chime in about where I'm at -- one problem was solved with a statistical classifier, but to bootstrap another, I ended up using keywords. It took a few hours to get a reasonable solution, and it leans more towards precision than recall, but it worked quickly!

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#66

Post by IanCal »

ffsm8 wrote: post #60
IanCal wrote: post #57

Earlier quoted context omitted.

No? But repeated words can impact simple nlp setups. I’m not sure what case you’re concerned about where added text impacts classification with an LLM but added words shouldn’t with a different pipeline. > And NLP stands for natural language processing. If the result didn't change after you've made changes to the input... It'd be a bug? No, I’d want my classifier to be unchanged by garbage words added. It likely will…

Prompt injection is about making the model do something else then specified. Adding words to the text to break the algorithm which does the NLP is more along the lines of providing 1 in a boolean field to break the system. And that's generally something you can mitigate to some degree via heuristics and sanity checking. Doing the same for LLMs is essentially impossible, because it's an effective black box, so you can…

If you don’t think this happens for simpler methods you’ve never deployed them. It’s the exact same problem on a classifier. Have you actually worked with these and are we discussing real world cases?

Re: SpaCy: Industrial-Strength Natural Language Processing (NLP) in Python

#67

Post by leobg »

coder68 wrote: post #62
leobg wrote: post #61

Earlier quoted context omitted.

If I have 1,000 labeled examples for a classification task, I’ll expand that into a training dataset using augmentation, and then finetune a small model like RoBERTa. It’s fast, cheap, accurate — and predictable. Others have had success with SetFit as the training framework and Ettin as the base model.

oh this seems like an interesting idea, what tactics do you use for augmentation? For my own use-case, I think I could reorder semantic chunks, or maybe randomly delete pieces, but curious what tactics you use! I have also considered training a small language model for synthetic data generation.

Yes, exactly. You want to randomize the parts that are irrelevant. For example, if you're classifying news articles, you may want to shorten them anyway. A human would be able to tell what category an article belongs to without reading the whole thing - so may do a combination of URL, headline, beginning, middle, and/or end. And if you do that, it's easy to turn one training example into 10 or more. You just vary the length of the individual parts.
Post reply on HN