Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speech and Language Processing (3rd ed. draft)

web.stanford.edu

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Speech and Language Processing (3rd ed. draft)

#21

Post by neverokay »

Al-Khwarizmi wrote: post #10
ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

This is an interesting question. As an NLP researcher who is in contact with companies that demand NLP applications, I think we are not there yet. For example, a company that wants to extract information from medical records cannot use solutions like GPT-4 or Claude that involve sending protected data to third parties in foreign jurisdictions. Modest local models (like 7B models) don't work so well for things like na…

From my experience, entity recognition is very good with gpt4. I recorded CNN for hours and it was able to identify and summarize all the commercials.

It’s magic stuff

Re: Speech and Language Processing (3rd ed. draft)

#22

Post by maebert »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

You're not entirely wrong.

10 years ago, I started an ML & NLP consulting firm. Back then nobody was doing NLP in production (SpaCy hadn't come out yet, efficient vector embeddings were not around, the only comprehensive library to do NLP was NLTK, which was academic and hard to run in prod).

I recently revisited some of our projects from back then (like this super fun one where we put 1 Million words into the dictionary [1]) and realized how much faster we could have done many of those tasks with LLMs.

Except we couldn't — the whole "in production" part would have made LLMs for the most minute tasks prohibitively expensive, and that is not going to change for a while, sadly. So, if you want to work something in prod that is not specifically an LLM application, this book is still super valuable.

[1] https://www.nytimes.com/2015/10/04/technology/scouring-the-w...

Re: Speech and Language Processing (3rd ed. draft)

#23

Post by [deleted] »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

[deleted]

Re: Speech and Language Processing (3rd ed. draft)

#24

Post by k8si »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

I suggest going through the exercise of seeing whether this is true quantitatively. Get a business-relevant NER dataset together (not CoNLL, preferably something that your boss or customers would care about), run it against Mistral/etc, look at the P/R/F1 scores, and ask "does this solve the problem that I want to solve with NER". If the answer is 'yes', and you could do all those things without reading the book or other NLP educational sources, then yeah you're right, job's finished.

Re: Speech and Language Processing (3rd ed. draft)

#25

Post by gillesjacobs »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

LLMs typically do worse on specific extractive and classification tasks than a finetuned BERT-large model, so no you actually can't replace everything with LLM calls with similar performance.

(Cf. BloombergGPT paper all financial benchmark tasks).

And that's not even taking into account inference cost, but that is a business case issue.

Re: Speech and Language Processing (3rd ed. draft)

#27

Post by chaxor »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

If you want to process 100M documents, you want to use the most performant and fastest option - which a 7b param model isn't going to be it.

Additionally, entity linking is an extremely common task, for which LLMs fail at pretty miserably (certainly if the dictionary is custom/private. Additional work must be done to somehow (!? Many options here ?!) perform EL.

So, in the end, making a silver corpus from an LLM may be an option for NER to train a much much smaller algorithm. But EL is _still_ not a 'plug and play' problem, and can actually be pretty difficult to do "well" (using the modern techniques of MHS, etc).

Re: Speech and Language Processing (3rd ed. draft)

#28

Post by esafak »

miraculixx wrote: post #15
ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

Reliability, is what you are missing. LLM are not reliable.

And cost effectiveness.

Re: Speech and Language Processing (3rd ed. draft)

#29

Post by hintymad »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

Take a Lexis Nexis as an example. They used to build sophisticated NLP pipelines to squeeze information out of legal documents: Part-of-Speech tagging, dependency parsing, NER, topic modeling, relation extraction, event extraction, summarization, and etc. They also spend millions of dollars on custom training just to expand the entity types and etc. The whole process is error prone, painful to maintain, and expensive. Similarly, McDonald must have struggled a lot building their automatic ordering system.

LLM must be a godsend for these companies. All of sudden, low-level tasks like POS can be eliminated. Tasks like NER have only limited use and the companies can enjoy orders more entity types almost for free. Tasks like intention slotting and topic modeling become trivial compared to the pre-LLM-era pipelines.

Re: Speech and Language Processing (3rd ed. draft)

#30

Post by abhgh »

ilaksh wrote: post #4

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…

You're correct except for the use-cases where one of these come into play:

A. Latency: for some systems, you need near real-time predictions. LLMs (today) are slow for that.

B. Cost: when the low dev. effort (for building and deploying an ML model) and low sample complexity (i.e. zero/few-shot) doesn't translate into proportionate monetary gains over what you pay for LLM usage.

C. Precision: when you want the model to reliably tell you when it doesn't know the correct answer. Hallucination is a part of it - but I think of this requirement as the broader umbrella of good uncertainty quantification. I think there are two reasons why this is worse for LLMs: (1) traditional ML models also suffer from this, but there are some well known ways for mitigation. For LLM's there is still no universal or accepted way to perform this reliably (2) the quality of generated language an LLM produces seems to be more likely to deceive you when it is wrong. I don't know how to scientifically think about this - maybe as LLMs proliferate people would build appropriate mental defenses?

There is also the practical problem of prompt transferability across LLMs: what works best for one LLM might not work well for another, and there is no systematic way to optimally modify the original prompt. This is painful in some setups where you're looking to be not locked-in. But I didn't put it in the list because this seems to be a problem for niche groups - everyone seems to be busy in getting stuff working with one LLM. Maybe this will become a larger issue later.

Post reply on HN