Speech and Language Processing (3rd ed. draft)

[dead]

I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My other assumption is that this wasn't true at all three years ago.

Maybe machine learning engineers want to explain what I am missing?

Well, I can see it as being useful to give aspiring ML researchers a survey and jumping off point for possibly digging deeper.

But I also think when they created that book originally it was about giving ML practitioners tools they would use.

It depends on what you are trying to do.

LLMs are powerful, but can be difficult to work with w.r.t. doing post processing or adding markup/annotations in the text. For example, asking it to label terms in a sentence it can produce varied output, e.g. sometimes listing "word or clause: description of the meaning" which is hard to parse in downstream tasks, or sometimes out of order.

I've also seen LLMs label split infinitives with the correct meaning at the preposition instead of the whole subclause. With NLP you would label either the start and span of the label (common) or the root of the subclause, depending on the application. The NLP approach would work for more complex nested and interconnected expressions where the structure is not flat text.

If you ask it to generate XML or HTML, it will usually invent its own markup, or sometimes generate invalid output. E.g. I've seen it output HTML using tags like `` in some cases.

Asking it to generate CoNLL-U (which it knew about from its response) -- it only outputted 3 columns as "in _ PREP" etc. which is not correct.

Asking `Can you lemmatize "She is going to the bank to get some money."` I get `"She go(ing) to bank(er) for money(y)"` which leaves out words, doesn't lemmatize some words correctly, and has a format that is difficult to parse/interpret reliably.

LLMs also have a limited context window, so if you are trying to process a large document, or have a large complex set of instructions (e.g. on how to label, process, or format the data) then it can lose those instructions and deviate from what you are doing.

LLMs are also susceptible to being guided by the input text as the prompt and input are taken together. Thus, if the text is talking about a different format or something else it can easily switch to that.

--

While NLP pipelines require more work to get right, they can often be more efficient computationally as they are often a lot smaller than the 7b parameters, or use other techniques that don't use ML/NNs.

You can build more custom pipelines by querying the different features (part of speech, lemma, lexical features, etc.) without having to reparse the data, and you can keep the annotations consistent across those different pipelines, e.g. when labelling, extracting and storing the data in a database for searching, etc. so a user can see all the places where a term is referenced in a given text.

This is an interesting question.

As an NLP researcher who is in contact with companies that demand NLP applications, I think we are not there yet. For example, a company that wants to extract information from medical records cannot use solutions like GPT-4 or Claude that involve sending protected data to third parties in foreign jurisdictions. Modest local models (like 7B models) don't work so well for things like named entity extraction yet. And furthermore, companies typically want some explanation and accountability for the results (especially in sensitive domains...) and sure, LLMs can explain, but you have no guarantee that the explanation isn't hallucinated. When I mention the possibility of hallucinations, companies typically balk and say that they prefer the classic way.

My (potentially biased) opinion is that classic NLP still has life left in it. For how long, I don't know. If small, locally-runnable LLMs get much better and more reliable, addressing the hallucination problems, what you mention might become largely true for most engineering applications.

Also note that beyond engineering, things like syntactic parsing are also useful for scientific pursuits too, and at the moment they seem to be out of reach of LLMs.