Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A PhD student's perspective on research in NLP in the era of LLMs

arxiv.org

Post reply on HN
Print view View original

41–50 of 55 posts

Re: A PhD student's perspective on research in NLP in the era of LLMs

#41

Post by rhdunn »

morkalork wrote: post #32
DougBTX wrote: post #7

Earlier quoted context omitted.

Do you strictly need that level of tokenisation precision to meet your high-level goals?

This is my first reaction as well. Talking about tokenization and POS tagging is getting lost in the weeds when one has goals like this: >I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collection, I don't want it to pick up a story where a detective is only mentioned in one paragraph. This more like N…

I also want to tag and highlight those parts of the document. For that, I need to know where the label starts and ends, which you can't really do when you don't have control over the tokens.

It's also hard to write custom inference/tagging rules, like in the case you mentioned w.r.t. Voldemort, if you don't know what the tokens look like.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#42

Post by chaxor »

vpastore wrote: post #23
dontupvoteme wrote: post #13

Are papers becoming blogs?

Only the one on arxiv. People shouldn’t take anything coming from that site seriously

This is an extremely wrong viewpoint. "Many extremely important* papers have been published on arxiv alone - no other publisher. Just off the top of my head, without looking, ELMo from allenAI (the paper that started the contextualized representations revolution for BERT/attention) is only on arxiv.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#43

Post by chaxor »

rhdunn wrote: post #40
nl wrote: post #11

Earlier quoted context omitted.

Have you tried Spacy? I find it substantially better than other tools as PoS tagger. Also worth noting the that your assertion that you need these features to classify genres isn't obviously true to me at all.

No I haven't. Thanks for the pointer. For detecting uses of nouns like werewolf/werewolves, or vampire/vampires, I at least need the lemma to avoid writing different cases or a regex for each noun. Likewise, lemmatization can be used to handle different spellings (e.g. vampyre, or were-wolf). Similarly for verbs. Lemmatization works best when it is coupled with part of speech tagging, so you avoid removing the -ing i…

I don't see why a simple TFIDF with ~10 LoC and a few minutes doesn't make this at least reasonably/crudely done.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#44

Post by chaxor »

rhdunn wrote: post #41
morkalork wrote: post #32

Earlier quoted context omitted.

This is my first reaction as well. Talking about tokenization and POS tagging is getting lost in the weeds when one has goals like this: >I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collection, I don't want it to pick up a story where a detective is only mentioned in one paragraph. This more like N…

I also want to tag and highlight those parts of the document. For that, I need to know where the label starts and ends, which you can't really do when you don't have control over the tokens. It's also hard to write custom inference/tagging rules, like in the case you mentioned w.r.t. Voldemort, if you don't know what the tokens look like.

Spacy is a decent suggestion here. They have pretty good ways of writing tagging rules.

All of this does seem to be extremely excessive to choose a book genre though. I would imagine the number of books after a simplistic clustering technique would be rather small to flip through, so I really don't understand the use case at all.

If you have very few books (few thousands) then you can apply more fine grained analyses in reasonable amounts of computation, such as contextualized embedding methods. But if the point is to select a book, there no real benefit since the simple 2 second term frequency methods would narrow choices down to only a few books.

If you have billions of books, contextualized embeddings become quite expensive to produce and use (several weeks or months of processing, petabytes of storage, etc), so it's not really feasible as an individual, But the extra querying capability does help narrow the large set down.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#45

Post by hawski »

viksit wrote: post #3
rhdunn wrote: post #2

I've been interested in NLP for tagging stories based on topics and themes (detectives, werewolves, murder mystery, etc.), so need accurate disambiguation of parts of speech and ways of detecting uses of metaphore, similies, etc. to describe those. I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collec…

at this point the gpt api will do all of that for you! or you can use the embeddings api and build your own systems.

I'm sure GPT will cover more ground much easier, but aren't LLMs in general much more resource intensive? It seems they will remain expensive to run for quite some time.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#47

Post by wahahah »

agalunar wrote: post #38
est31 wrote: post #31

Earlier quoted context omitted.

The particular headline pattern is used a lot and 99% of the time it indeed is at the front of opinion pieces written by individuals. But this time, it's different. In the syntax, in this specific form, the plural and the singular come out the same. E.g. in the sentence "I accompanied my friend to his parent's house", it can be either the house of his single mom/dad or the house of his two parents living together.

Like @bdsa points out with their example, the singular and plural are actually spelled differently, viz. "parent's house" and "parents' house", despite being pronounced the same way. https://en.wikipedia.org/wiki/English_possessive#Nouns_and_n...

I pronouce the latter as ~parent-siz; having them the same never sat right with me.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#48

Post by teleforce »

According to the article, the original research on language model was kick started by Claude Shannon's early contributions in Markov chain model of English words.

If you are in the field of Information and Communication Technology (ICT) there are hardly any area in the field which their fundamentals do not have Shannon's hands in it.

Leonard Kleinrock once remarked that he has to focus on the exotic queuing theory field that later leads to the packet switching and then Internet because most of the fundamentals problems in electrical and computer engineering (older version of ICT) have already been solved by Shannon.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#49

Post by mschuetz »

vpastore wrote: post #23
dontupvoteme wrote: post #13

Are papers becoming blogs?

Only the one on arxiv. People shouldn’t take anything coming from that site seriously

Many of the best papers appear an Arxiv first. In some fields, it is customary to put your preprint on Arxiv before/during the submission to the peer reviewed venue.

Arxiv is vital for quickly developing research fields.

Re: A PhD student's perspective on research in NLP in the era of LLMs

#50

Post by est31 »

agalunar wrote: post #38
est31 wrote: post #31

Earlier quoted context omitted.

The particular headline pattern is used a lot and 99% of the time it indeed is at the front of opinion pieces written by individuals. But this time, it's different. In the syntax, in this specific form, the plural and the singular come out the same. E.g. in the sentence "I accompanied my friend to his parent's house", it can be either the house of his single mom/dad or the house of his two parents living together.

Like @bdsa points out with their example, the singular and plural are actually spelled differently, viz. "parent's house" and "parents' house", despite being pronounced the same way. https://en.wikipedia.org/wiki/English_possessive#Nouns_and_n...

Oh that's interesting, thanks! Not a native speaker but I dimly remember hearing about this in english class, thanks for reminding.

I guess then the PhD student is indeed grammatically a singular then. It can still refer to a PhD student in general though instead of a particular one.

https://www.rit.edu/ntid/sea/processes/articles/grammatical/...

Post reply on HN