Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Natural language processing to identify grammar in a text?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 25 posts

Ask HN: Natural language processing to identify grammar in a text?

#1

Post by davearms »

Apologies for the clunky question. We have a growing base of adult English students. Our teaching methodology is content-first, basically we find any material of interest to our students, based on their interests, sector and language goals and build lesson plans and discussions around that 'centerpiece'. A lot of the work is curation and creating reusable discussion questions.

I have been searching for a tool that can scan a paragraph and extract the grammar tenses and features (past simple, present continuous, passive voice, indirect question) as it's a recurring question with our students. We have tools to tell us the approximate level, suggested vocabulary and word count, but does this even exist (yet?). Thank you in advance.

Re: Ask HN: Natural language processing to identify grammar in a text?

#2

Post by dpezely »

There are several topics intertwined with solutions you seek: There's parts of speech (PoS) tagging, reducing to Lemma form, identifying end of sentence, etc.

After having faced a similar learning curve, I put what I know into a lengthy document[0] written in 2018 based upon explorations over 2016-17. That will get you deployed and operational quickly by following just the final section. The first section explains key concepts using conventional ideas as the means of introducing NLP jargon. In between covers theory and practice for getting the most out of any tool you're likely to use in the end.

More general tools are probably available today, such as add-ons for Elasticsearch. I'd start looking there. Interesting items came up when searching ddg for: NLP elasticsearch.

[0] http://play.org/articles/introduction-to-natural-language-pr...

Re: Ask HN: Natural language processing to identify grammar in a text?

#4

Post by bryanrasmussen »

dpezely wrote: post #2

There are several topics intertwined with solutions you seek: There's parts of speech (PoS) tagging, reducing to Lemma form, identifying end of sentence, etc. After having faced a similar learning curve, I put what I know into a lengthy document[0] written in 2018 based upon explorations over 2016-17. That will get you deployed and operational quickly by following just the final section. The first section explains ke…

maybe also try searching for nlp Lucene

There's also a pretty good book https://lingpipe-blog.com/2008/06/12/book-building-search-ap...

of course the more general tools available today notice applies in regard to this book as well.

Re: Ask HN: Natural language processing to identify grammar in a text?

#5

Post by pattusk »

I don't think there's an out-of-the box library for things like the detection of passive voice or indirect questions. But you should be able to build something that would let you do it based on the basic NLP toolkit: dependency parsing, POS tagging, lemmatization, named entity recognition.

I suggest you check out Spacy [0] for a quick and easy to use Python library providing the above features. The software produced by the Stanford NLP Group is also great [1].

If you do not want to get your hands dirty with code, there are a number of API providers that will offer you the same as the above libraries (TextRazor, Rosette Text Analytics...)

[0] https://spacy.io/

[1] https://nlp.stanford.edu/software/

Re: Ask HN: Natural language processing to identify grammar in a text?

#6

Post by unhammer »

On recent debians/ubuntu, PoS tagging is just one apt away:

    $ sudo apt install -y apertium-eng

    $ echo "I have been searching for a tool that can scan a paragraph" |apertium eng-disam|grep -v '^;'
    ""
            "prpers" prn subj p1 mf sg
    ""
            "have" vbhaver inf
            "have" vbhaver pres
    ""
            "be" vbser pp
    ""
            "search# for" vblex ger SELECT:177
    ""
            "a" det ind sg
    ""
            "tool" n sg
    ""
            "that" cnjsub
            "that" prn dem mf sg
            "that" prn rel an mf sp
    ""
            "can" vbmod pres SELECT:281
    ""
            "scan" vblex inf SELECT:140
    ""
            "a" det ind sg
    ""
            "paragraph" n sg
    ""
            "." sent
(grepping out lines with ; since they just show what was not removed by the disambiguator, whereas SELECT/REMOVE are just trace info saying what rules applied. If there are multiple indented lines, then the disambiguator didn't manage to fully disambiguate the analysis.)

If you want to e.g. mark passive, it's easy to write a Constraint Grammar rule to do this. Put the following into rules.cg3:

    DELIMITERS = sent ;
    ADD (&PASSIVE) ("be") # Add the tag "&PASSIVE" to the word with lemma "be"
    IF
    (1* (pp)             # There is a participle to the right
     BARRIER (*) - (adv) # with nothing in between except perhaps adverbs
    );
and pipe it in after the above pipeline:

    $ echo "The paper is not signed by me" |apertium eng-disam |grep -v '^;'|vislcg3 -g rules.cg3
    ""
            "the" det def sp
    ""
            "paper" n sg
    ""
            "be" vbser pres p3 sg &PASSIVE
    ""
            "not" adv
    ""
            "sign" vblex pp
            "sign" vblex past
            "signed" adj
    ""
            "by" pr SELECT:470
    ""
            "prpers" prn obj p1 mf sg
    ""
            "." sent
( https://wiki.apertium.org/wiki/Constraint_Grammar for more info on CG )

Re: Ask HN: Natural language processing to identify grammar in a text?

#8

Post by Timpy »

This isn't really a programming solution but you should check out this webapp:

http://www.hemingwayapp.com/

It may not be the exact thing you're looking for but it can probably be helpful to your students.

I would also look at Python NLTK. I've only dabbled in the toolkit, so I'm not sure if it has what you're looking for exactly, but it's worth a look.

http://www.nltk.org/

Re: Ask HN: Natural language processing to identify grammar in a text?

#9

Post by Tainnor »

unhammer wrote: post #6

On recent debians/ubuntu, PoS tagging is just one apt away: $ sudo apt install -y apertium-eng $ echo "I have been searching for a tool that can scan a paragraph" |apertium eng-disam|grep -v '^;' " " "prpers" prn subj p1 mf sg " " "have" vbhaver inf "have" vbhaver pres " " "be" vbser pp " " "search# for" vblex ger SELECT:177 " " "a" det ind sg " " "tool" n sg " " "that" cnjsub "that" prn dem mf sg "that" prn rel an m…

That rule is really rather limited. More in general, I doubt you can properly recognise passive constructions without doing constituent parsing, passive is really a syntactic construction, not a morphological one.
Post reply on HN