An Efficient Way to Extract the Main Topics from a Sentence

An Efficient Way to Extract the Main Topics from a Sentence
thetokenizer.com

The article gives an example which I find a bit confusing.

>I ran it on this sentence -

> “Swayy is a beautiful new dashboard for discovering and curating online content.”

>And got this result -

> This sentence is about: Swayy, beautiful new dashboard, online content

That misses "discovering" and "curating", which I think are the most important parts of that sentence.

So you're just identifying NPs and VPs in a sentence? So lets say I run your program, and I get NPs "Instagram" and "Facebook", and the VP "acquired." The question is, who did what to whom? Did Facebook acquire Instagram, or did Instagram acquire Facebook?

Second, I think you're way over-emphasizing the supposed slowness of CFG parsing. Yes, the complexity is O(n^3) in the length of the sentence, but in practice, n is usually small. Modern statistical PCFG parsers are fast.

It's a dashboard for online content. That pretty much implies the picking and finding of said online content to be displayed on the dashboard.

Shlomib, you might be interested in SHRDLU [1][2] if you are not aware of it. It was developed by Terry Winograd [3] for his dissertation [4] at MIT. It is a natural language understanding [5] parser that allows you to interact with a small world of 3D solids.

I think you will find the paper interesting, because it goes into detail on sentence structure and associated parsing.

Here is a sample dialogue with the program, taken from Wikipedia:

  Person:   Pick up a big red block.
  Computer: OK.
  Person:   Grasp the pyramid.
  Computer: I don't understand which pyramid you mean.
  Person    (changing their mind): Find a block which is taller than the one you are holding and put it into the box.
  Computer: By "it", I assume you mean the block which is taller than the one I am holding.
  Computer: OK.
  Person:   What does the box contain?
  Computer: The blue pyramid and the blue block.
  Person:   What is the pyramid supported by?
  Computer: The box.
  Person:   How many blocks are not in the box?
  Computer: Four of them.
  Person:   Is at least one of them narrower than the one which I told you to pick up?
  Computer: Yes, the red cube.

[2] http://hci.stanford.edu/winograd/shrdlu/

[3] https://en.wikipedia.org/wiki/Terry_Winograd

[4] http://hci.stanford.edu/winograd/shrdlu/AITR-235.pdf

[5] https://en.wikipedia.org/wiki/Natural_language_understanding