It seems to not properly handle embedded HTML; using my feed (http://www.dp.cx/blog/rss.xml), look at the story titled "The Difficulty of Parsing the Web" and notice the box that is rendered.
Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
51–60 of 68 posts
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#52Hello. I recently finished my thesis for my MS CS degree. My thesis is about automatic summarization. It undergoes research, defense, and I think its result is good enough for me. It uses statistical approach and machine learning. My main issue about it is not the summarization part, but the text extraction part. I can't seem to extract article in a web page well enough. I'm using boilerpipe ( https://code.google.com…
Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thank…
Look up text rank. Essentially it amounts to representing text as a graph with sentences as nodes and some sort of similarity measure as edge weights. You then run page rank on that graph.
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#53Earlier quoted context omitted.
Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thank…
Look up text rank. Essentially it amounts to representing text as a graph with sentences as nodes and some sort of similarity measure as edge weights. You then run page rank on that graph.
Interesting. Thanks! :)
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#54Earlier quoted context omitted.
Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thank…
Hi, I recommend this following research: http://www.cs.cmu.edu/~nasmith/LS2/das-martins.07.pdf http://www.aclweb.org/anthology-new/W/W03/W03-1204.pdf I still have other papers, but those can be a good starting point. In my thesis, NLP is done via statistical approach. It learns from it's previous summaries does have somewhat learning. I don't see any problems combining NLP with Machine Learning. Can you elaborate on…
Thanks, these look like exactly what I was looking for.
Re NLP, I meant to say NLP from sentiment analysis perspective (not sure if that's the right way to put it). So my question was whether you had to extract meaning of one or few related sentences or only process the text in a statistical manner (which you answered).
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#55Not bad for a weekend, but http://www.tldrstuff.com does a much better job. Especially where the sentences don't break on . like where J. R. R. Tolkien is concerned.
And The TLDR Plugin works with HTML and on all western languages.
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#56As as cofounder of http://tldr.io this confirms our vision that for now (and for many years) only people can perform such a hard task like summarizing.
People are inherently biased. It's impossible for any person to read news and not inject their own personal tendencies into a summary they write.
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#57Generated 5 Sentence Summary for http://www.businessinsider.com/why-marissa-mayer-bought-a-30... Back in March, Yahoo bought a startup called Summly for $30 million. Before Yahoo shut it down, Summly was a news aggregation app for smartphones. According to Summly's own Web site, the technology behind the app was "built" by an organization called "SRI International," not by the startup's employees. And indeed, inside…
http://TLDRstuff.com comes back with:
Why Marissa Mayer Bought A $30M Startup - Business Insider: The deal got a lot of attention because Summly's CEO is 17-year-old Nick D'Aloisio. Acquiring Summly seems to have been an almost incidental side effect of a deal Yahoo made with SRI for a piece of "summarization technology." Until Yahoo bought it, SRI International held equity in Summly. The job of implementing this technology at Yahoo will not be given to anyone from Summly, including its young CEO.
Notice that the version from TLDR Stuff actually has the Answer in it. "Incidental side effect of a deal Yahoo make with SRI"
It also tells you that they aren't interested in the young CEO.
This is possible because it is not a keyword density Algo, the core technology called Liquid Helium is a Language Heuristics Engine and it can put weight on which sentences are Causality, and which are Subject matter. This creates a version of the text that tells you Who What Why and if there is still space, How. You can't do that with just a KeyWord density or where in the article is this system.
Summly claimed to have that tech, SRI has some of it, but what they really have is a nice Concept Tree, and sentence parser.
A far cry from a system that knows which points are important, not just which points are most talked about, because as you see in this Business insider article the important part isn't "what is summly" or "who is nick" or "Who is mayer" it is "Why did Yahoo do this" and that is captured in the TLDRSTuff/Stremor version, and not in TLDRizer.
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#58I've got an IOException while trying to summarize http://matt-welsh.blogspot.com/2013/04/running-software-team... And a different one for http://googleblog.blogspot.com I guess you should put more effort in your html parser. Try Apache Tika, perhaps.
okay, added a fix to try and extract article text from non-feed urls. try http://tldrzr.herokuapp.com/tldr/?feed_url=http://matt-welsh... :)
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#59I really like this. Here's an example of one of PG's essays run through the algorithm: [ http://paulgraham.com/startupideas.html ] The most important thing to understand about paths out of the initial idea is the meta-fact that these are hard to see. Empirically, the way to have good startup ideas is to become the sort of person who has them. If you know a lot about programming and you start learning about some other…
Thanks for the idea. Try this :) http://tldrzr.herokuapp.com/tldr/?feed_url=http://www.paulgr...
Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)
#60Now the url can be a page, I try to extract the article text using boilerpipe. :) Also added a simple GET endpoint for linking. Try this summary of PG's "Writing and Speaking" essay: http://tldrzr.herokuapp.com/tldr/?feed_url=http://www.paulgr...