Machine Learning is evolving very quickly. What are the state of the art techniques to extract specific facts from the text automatically?
Are there any open source projects focused on this task that you could recommend?
Ask HN: How to use Machine Learning to extract facts from the text?
1–10 of 23 posts
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#2Looking at SyntaxNet from Google. If there are other candidates worth looking at - please kindly let me know.
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#3How do you define a fact?
As far as I understand it symbolic AI back in the 80s was building a massive web of facts or "truths" that would be used to create a general AI. They eventually ended up generating a bunch of contradictions.
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#4Looking for something similar for .pdf academic papers in my field but nothing really useful to automatise the extraction process exists, so the best path is still to extract data manually, homogenise data in a standard protocol, fed ML algos. Once a data protocol becomes a widespread standard and maybe a ISO or similar, there is a chance automated extraction will work at the finest level, as necessary for complex information.
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#5Looking at SyntaxNet from Google. If there are other candidates worth looking at - please kindly let me know. https://github.com/tensorflow/models/tree/master/syntaxnet
Depends on the corpus. If your problem fits a conLL task, you can read lots of papers about it. If you can build on an existing wikipedia entity/relation graph, dictionary, gazzette that's a big boost. For academic research papers, look at citations for your input stream then SVM tf-idf bigrams. If sentiment/quality analysis, that's another tack
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#6How do you define a fact? As far as I understand it symbolic AI back in the 80s was building a massive web of facts or "truths" that would be used to create a general AI. They eventually ended up generating a bunch of contradictions.
Very good question. I don't have good knowledge yet how to model this correctly.
Currently I imagine that for given domain I can create text parser, which would extract facts in standard formats. The example could be: "object predicate subject". And then use facts mapped to documents for relevant domain search and validation of some basic statements in other documents.
Not all statements require validation, I can focus only on those which have high confidence in being parsed correctly.
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#7Looking at SyntaxNet from Google. If there are other candidates worth looking at - please kindly let me know. https://github.com/tensorflow/models/tree/master/syntaxnet
Depends on the corpus. If your problem fits a conLL task, you can read lots of papers about it. If you can build on an existing wikipedia entity/relation graph, dictionary, gazzette that's a big boost. For academic research papers, look at citations for your input stream then SVM tf-idf bigrams. If sentiment/quality analysis, that's another tack
Right now I'm hoping to extract entities and relations between them to use as facts for relevant domain specific search and validation.
Reading up the articles, YCombinator included: https://blog.ycombinator.com/how-to-get-into-natural-languag...
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#8Looking for something similar for .pdf academic papers in my field but nothing really useful to automatise the extraction process exists, so the best path is still to extract data manually, homogenise data in a standard protocol, fed ML algos. Once a data protocol becomes a widespread standard and maybe a ISO or similar, there is a chance automated extraction will work at the finest level, as necessary for complex in…
I found these two projects so far:
https://cloud.google.com/natural-language/
https://www.textrazor.com/demo
Do you expect the approach of extracting entities and relations between them to be limited? I hope that it can be boosted for specific domain with predefined entities and facts structures.
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#9Looking for something similar for .pdf academic papers in my field but nothing really useful to automatise the extraction process exists, so the best path is still to extract data manually, homogenise data in a standard protocol, fed ML algos. Once a data protocol becomes a widespread standard and maybe a ISO or similar, there is a chance automated extraction will work at the finest level, as necessary for complex in…
I found these two projects so far: https://cloud.google.com/natural-language/ https://www.textrazor.com/demo Do you expect the approach of extracting entities and relations between them to be limited? I hope that it can be boosted for specific domain with predefined entities and facts structures.
I do not know how these tools work today but I can imagine a dictionary becoming standard for each given field and being provided with every future .pdf paper as a companion in the form of a .csv sparse matrix or similar format. Such dictionaries will be put forward by researchers for exchanging data in the smoothest way, agreed as a standard after some time and then forced into adoption worldwide. We will get there for sure, some fields before others, some industries have started already (aeronautics, automotive).
Re: Ask HN: How to use Machine Learning to extract facts from the text?
#10What exactly is a fact? There's no easy answer to that question, particularly with natural rather than formal languages. 'Facts' and statements depend on context. The meaning of a natural language statement usually is derived from these layers building on each other:
- syntax (the structure of a sentence)
- semantics (the isolated meaning of a sentence)
- pragmatics (the meaning of a sentence in context)
Anaphora (references to previous sentences or concepts) can be particularly nasty in this context.
Depending on the task at hand chunk parsing could be a good first take at finding relevant phrases from unstructured textual data. There are numerous libraries to accomplish that, for English and other Indo-European languages at least.