31–38 of 38 posts
It was the first edition of this book that, when I read it 20 years ago, got me "hooked" on computer science as a science rather than as just a fun thing to do (which I had been doing for several years prior). When I met Dan Jurafsky, 10 years later, I definitely felt none of that disappointment that often goes with meeting one of your "childhood heroes" and thanked him for writing the book and the impact it had on m…
I've been downloading and reading various chapters of this book for what seems like years now. I'd love a hard copy. I know it says "When will the whole book be finished? Don't ask." But it seems like when the missing Chapter 12 is added ("We also expect to release Chapter 12 soon in an updated release.") it would be a great time to print it. I can hope....
I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…
LLMs typically do worse on specific extractive and classification tasks than a finetuned BERT-large model, so no you actually can't replace everything with LLM calls with similar performance. (Cf. BloombergGPT paper all financial benchmark tasks). And that's not even taking into account inference cost, but that is a business case issue.
I'm not sure what BloombergGPT has to do with LLMs vs non-LLMs; BloombergGPT is an LLM [2], and it defeating other LLMs on financial benchmarks doesn't prove much about large language models other than "LLMs can be trained to be better at specific tasks."
1: https://arxiv.org/abs/2005.14165
2: https://www.bloomberg.com/company/press/bloomberggpt-50-bill...
I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…
LLMs typically do worse on specific extractive and classification tasks than a finetuned BERT-large model, so no you actually can't replace everything with LLM calls with similar performance. (Cf. BloombergGPT paper all financial benchmark tasks). And that's not even taking into account inference cost, but that is a business case issue.
I almost believe that if I know how make an LLM prompt and how to make an API call to OpenAI, Mistral, Claude 3, or together.ai, then as an application programmer, I can skip this whole book. I see people posting project specifications asking for NLP, named entity extraction, etc. But most things in those jobs look like they could be handled by an LLM, possibly even smaller than 7b, and probably more robustly. My oth…
Need to use CRFs for NER? Test on your own validation set Need to use LLMs for NER? Test on your own validation set
The validation set should be made up of carefully curated samples that you've seen errors on, edge cases, just like Test-driven Development, but at a much larger scale. Assessing LLMs by demo is a horrible habit that we've taken to that needs to be changed.
Earlier quoted context omitted.
LLMs typically do worse on specific extractive and classification tasks than a finetuned BERT-large model, so no you actually can't replace everything with LLM calls with similar performance. (Cf. BloombergGPT paper all financial benchmark tasks). And that's not even taking into account inference cost, but that is a business case issue.
BERT is a language model! It was considered a "large" language model for its time, and it's even based on transformers. It's a very small language model by today's standards (340MM params), and is encoder-only instead of decoder-only, but trying to draw a hard line between "BERT" and "LLMs" is more about parameter count than capabilities — in fact, the original GPT-3 paper benchmarked GPT-3 against finetuned BERT-lar…
When I used "specific task", I meant specialized, domain specific tasks like financial sentiment and event extraction in which I hold a PhD. As a matter of fact, for Fiqa SA finetuned Roberta scores 88% F1 while BloombergGPT scores 75% F1. [1] Still very impressive for zero/few shot learner, but depending on data availability, performance targets and inference cost tradeoffs, it might not need your meds.
My point was "small" masked encoder transformer LMs like BERT can still hold their own on narrow domain tasks. And what OP claims that all NLP is solved by prompting a general purpose LLM service is simply inaccurate.
Earlier quoted context omitted.
BERT is a language model! It was considered a "large" language model for its time, and it's even based on transformers. It's a very small language model by today's standards (340MM params), and is encoder-only instead of decoder-only, but trying to draw a hard line between "BERT" and "LLMs" is more about parameter count than capabilities — in fact, the original GPT-3 paper benchmarked GPT-3 against finetuned BERT-lar…
Of course BERT is an LM, I never claimed it wasn't. It is just much smaller than what is now termed "LLM" which is typically an extremely large generative decoder-only transformer for general multi-language (often instruction and chat tuned). I was training and finetuned dozens of LMs back when BERT-large was considered a GPU memory issue and I have finetuned hundreds LMs over my NLP research and engineering career.…
I hadn't read the financial paper you linked, it's very interesting! One odd bit I did notice was they set the gpt-4 temperature to 1.0, which is... not a great setting for analysis, and probably harmed the results somewhat. Typically you'd want a value much closer to 0 for that. But while a lower, more reasonable temperature setting would probably improve gpt-4's performance, I would still expect a finetuned LM to outperform larger models with just prompting on those kinds of narrow domains, especially once cost is a factor.
It's somewhat surprising to see how bad Bloomberg-GPT was... Even gpt-4 trounced it on every published metric, and it wasn't trained for finance tasks specifically. The bitter scaling lesson, I suppose.