31–40 of 55 posts
A PhD Student's Perspective... ~20 authors.
Earlier quoted context omitted.
The problem with GPT and other LLMs is that they don't tokenize words at a word or morpheme level, it's just blocks of up to 4 characters, so you get tokens like `!"` instead of two separate tokens. -- That makes it harder to write custom tools on top of, unlike e.g. the output/model of things like the universaldependencies project.
Do you strictly need that level of tokenisation precision to meet your high-level goals?
>I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collection, I don't want it to pick up a story where a detective is only mentioned in one paragraph.
This more like NLU than an NLP problem isn't it? It's like tracking how much of a Harry Potter book contains Voldemort content without knowing ahead of time that he may be referred to as He Who Must Not Be Named, You-Know-Who, The Dark Lord and so on. One would have to first identify the thing you're interested in, then learn when characters/the author invent new ways to refer to it, and carry all those forwards to find new instances. Fun!
A PhD Student's Perspective... ~20 authors.
The particular headline pattern is used a lot and 99% of the time it indeed is at the front of opinion pieces written by individuals. But this time, it's different. In the syntax, in this specific form, the plural and the singular come out the same. E.g. in the sentence "I accompanied my friend to his parent's house", it can be either the house of his single mom/dad or the house of his two parents living together.
Are papers becoming blogs?
this paper doesn't appear to have been edited by any reputable journal, so take its authenticity at face value...
A PhD Student's Perspective... ~20 authors.
The particular headline pattern is used a lot and 99% of the time it indeed is at the front of opinion pieces written by individuals. But this time, it's different. In the syntax, in this specific form, the plural and the singular come out the same. E.g. in the sentence "I accompanied my friend to his parent's house", it can be either the house of his single mom/dad or the house of his two parents living together.
https://en.wikipedia.org/wiki/English_possessive#Nouns_and_n...
A PhD Student's Perspective... ~20 authors.
I've been interested in NLP for tagging stories based on topics and themes (detectives, werewolves, murder mystery, etc.), so need accurate disambiguation of parts of speech and ways of detecting uses of metaphore, similies, etc. to describe those. I also want to be able to assess how much of the text is about a given topic, so that if I'm interested in reading a detective story from e.g. the Project Gutenberg collec…
Have you tried Spacy? I find it substantially better than other tools as PoS tagger. Also worth noting the that your assertion that you need these features to classify genres isn't obviously true to me at all.
For detecting uses of nouns like werewolf/werewolves, or vampire/vampires, I at least need the lemma to avoid writing different cases or a regex for each noun. Likewise, lemmatization can be used to handle different spellings (e.g. vampyre, or were-wolf). Similarly for verbs.
Lemmatization works best when it is coupled with part of speech tagging, so you avoid removing the -ing in adverbs for example.
Part of speech tagging also helps avoid incorrect labeling, such as not tagging 'bit' in "a bit is a single binary value" as the verb "to bite".
That's for the simple case.
Then there are more complex cases, like generalizing "[NP] was bitten by the vampire.", where NP can be a personal pronoun (he, she, etc.) or a name. There can also be other ways to say the same thing, e.g. "The vampire bit [NP] neck." where NP is now the object form (his, her, etc.) not the subject form. With UniversalDependencies or similar style dependency relations, you could match and label sentence fragments of the form "verb=bite, nsubj=vampire, obj=NP" (like in the first sentence) and "verb=bite, nsubj:pass=NP, obj=vampire" (like in the second sentence).
Without NLP, it becomes even harder to detect split variants like "cut off his head" and "cut his head off", which are the same thing written in different ways. I want to detect things like that and label the entire fragment "beheading", including other noun phrase variants.
With more advanced NLP features -- like coreference resolution (resolving instances of he/she/etc. to the same person), and information extraction (e.g. Dracula is a vampire) -- it would be possible to tag even more sentences and sentence fragments.