Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

tldrzr.herokuapp.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#41

Post by peter_l_downs »

mohaps wrote: post #12
peter_l_downs wrote: post #10

Awesome! Looks super similar to an old sideproject of mine, www.bookshrink.com. The algorithm's different -- yours is aimed more towards summaries, while mine was aimed at sentence importance.

yeah, I'm working on adding more summarizer algorithms. I've been thinking on the lines of weighing up rhetorical questions, weighing down exclamation mark (cheap sarcasm detection) etc.

I experimented with giving bonuses to proper nouns and verbs, as well as giving a slight advantage to shorter sentences.

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#43

Post by bambax »

mohaps wrote: post #21
bambax wrote: post #15

Tried the rss feed from my blog and got an NPE: http://blog.medusis.com/rss Does it expect a specific format?

try now. the blog.medusis.com/rss link works now. Thanks for the feedback. since this grabs the page text (when no rss text is found) a lot of junk like copyright notices etc. shows up in summary. Will have to add some logic to scrub those. It also behaves horribly with code snippets.

Excellent, thanks, it does work now.

So what does it do exactly? It seems to extract some sentences more or less at random from the text...?

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#44

Post by hayksaakian »

Charlesmigli wrote: post #38

As as cofounder of http://tldr.io this confirms our vision that for now (and for many years) only people can perform such a hard task like summarizing.

It seems like your comment was well intended, but you come off as a bit presumptuous.

The problem TODAY is not whether or not a computer can summarize, but rather to what extent we as humans are satisfied with the computer's summary.

In some cases a dumb summary is good enough (first 200 characters for example). Given this baseline, and a target (human summary), you have to admit it's really an incremental process.

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#45

Post by mohaps »

hayksaakian wrote: post #44
Charlesmigli wrote: post #38

As as cofounder of http://tldr.io this confirms our vision that for now (and for many years) only people can perform such a hard task like summarizing.

It seems like your comment was well intended, but you come off as a bit presumptuous. The problem TODAY is not whether or not a computer can summarize, but rather to what extent we as humans are satisfied with the computer's summary. In some cases a dumb summary is good enough (first 200 characters for example). Given this baseline, and a target (human summary), you have to admit it's really an incremental process.

well put, hayksaakian. Also, never underestimate the built in auto-correct of the human mind :) There will always be a market for expert-curated approaches, but sometimes it's just cheaper to algorithmically "Crunch" it. Sometimes RAIN MAN counting toothpicks is enough, but when you need Ramanujam... :D

Also, do keep in mind... this is 2 hrs worth of coding time late on a sunday night. I don't have a CS degree, just a utilitarian/curious programmer who sometimes is stupid enough not to realize how hard a problem I'm tackling. :) Someone better qualified can do a much better job. Sometimes "just good enough" is good enough! :)

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#46

Post by mohaps »

bambax wrote: post #43
mohaps wrote: post #21

Earlier quoted context omitted.

try now. the blog.medusis.com/rss link works now. Thanks for the feedback. since this grabs the page text (when no rss text is found) a lot of junk like copyright notices etc. shows up in summary. Will have to add some logic to scrub those. It also behaves horribly with code snippets.

Excellent, thanks, it does work now. So what does it do exactly? It seems to extract some sentences more or less at random from the text...?

the algo is listed at the bottom of the home page. Will be opensourcing this code soon.

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#47

Post by MojoJolo »

Charlesmigli wrote: post #38

As as cofounder of http://tldr.io this confirms our vision that for now (and for many years) only people can perform such a hard task like summarizing.

I agree with "for now" but not "for many years". Right now, most or all automatic summarizers are doing extraction. Which is just lifting sentences from the original article itself. It is different from the human perception of summary, which is abstraction. That uses the most important parts of the article and paraphrase it for easy reading.

Right now, abstraction or paraphrasing is hard to do by a computer. But I think and hopefully it will be possible in few years time. There are various open source and academic tools that can do some pretty good NLP. I'm looking into Apache OpenNLP, and WordNet. I'm hoping for 2 or 3 years time.

BTW, I have an app similar to your tldr.io. Check my HN comment (https://news.ycombinator.com/item?id=5523770) for more info about it. ;)

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#48

Post by midko »

MojoJolo wrote: post #25

Hello. I recently finished my thesis for my MS CS degree. My thesis is about automatic summarization. It undergoes research, defense, and I think its result is good enough for me. It uses statistical approach and machine learning. My main issue about it is not the summarization part, but the text extraction part. I can't seem to extract article in a web page well enough. I'm using boilerpipe ( https://code.google.com…

Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thanks

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#49

Post by romain_g »

mohaps wrote: post #6
sherjilozair wrote: post #4

Earlier quoted context omitted.

This misses the most important line in the article: Acquiring Summly seems to have been an almost incidental side effect of a deal Yahoo made with SRI for a piece of "summarization technology".

the algo is still kinda "dumb". It basically tries to get top N keywords - most frequent non stopwords, stems them and goes through the sentences looking for which of them (upto max summary length) contain the keywords. I'll keep whittling at it over nights/weekends to see if I can make it more "semantically aware" edit: some work is needed on the tokenization also. currently, I don't preserve non-period punctuation.

You might want to look into topic modeling to extract the most meaningful topics (set of co-occuring words). It could greatly improve your results compared to the keyword approach. Interesting initiative, keep us posted !

Re: Show HN: TL;DRizer - an algorithmic summarizer webapp/api in java (weekend hack)

#50

Post by MojoJolo »

midko wrote: post #48
MojoJolo wrote: post #25

Hello. I recently finished my thesis for my MS CS degree. My thesis is about automatic summarization. It undergoes research, defense, and I think its result is good enough for me. It uses statistical approach and machine learning. My main issue about it is not the summarization part, but the text extraction part. I can't seem to extract article in a web page well enough. I'm using boilerpipe ( https://code.google.com…

Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thank…

Hi, I recommend this following research: http://www.cs.cmu.edu/~nasmith/LS2/das-martins.07.pdf http://www.aclweb.org/anthology-new/W/W03/W03-1204.pdf

I still have other papers, but those can be a good starting point.

In my thesis, NLP is done via statistical approach. It learns from it's previous summaries does have somewhat learning. I don't see any problems combining NLP with Machine Learning. Can you elaborate on this?

Post reply on HN