Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacker News "Submit" functionality needs a Digg-like duplicate alert

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 17 posts

Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert

#3

Post by brk »

Usually when I've submitted a dupe story, if it already exists it just adds a vote to the pre-existing story.

Where the logic either breaks (or allows people to subvert it) is where different URL's can get you to the same story. Often times URL's contain some superfluous flags that don't change the content served, but just serve to log some referrer or layout type data (I'm sure everyone reading this site gets how this works). Adding, removing, or changing any of this data seems to pretty much break or confuse whatever dupe-detector logic exists.

Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert

#4

Post by jharper »

brk wrote: post #3

Usually when I've submitted a dupe story, if it already exists it just adds a vote to the pre-existing story. Where the logic either breaks (or allows people to subvert it) is where different URL's can get you to the same story. Often times URL's contain some superfluous flags that don't change the content served, but just serve to log some referrer or layout type data (I'm sure everyone reading this site gets how th…

Human dupe-detection would be an excellent extension to this process.

Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert

#6

Post by brk »

jharper wrote: post #4
brk wrote: post #3

Usually when I've submitted a dupe story, if it already exists it just adds a vote to the pre-existing story. Where the logic either breaks (or allows people to subvert it) is where different URL's can get you to the same story. Often times URL's contain some superfluous flags that don't change the content served, but just serve to log some referrer or layout type data (I'm sure everyone reading this site gets how th…

Human dupe-detection would be an excellent extension to this process.

Are you suggesting that new submissions route through Mechanical Turk?

LOL.

Once approach might be that when humans detect dupes, they could be reported. Click the "dupe" link, specify the URL(s) of the dupe(s), and submit. The oldest submission "wins", and the data could be used to train a bayesian dupe detector. I imaging that you could start with a URL text match (it's the ends of the string that tend to be different), along with a check of the for the supposedly dupe page, and maybe the first 128 characters of the story text or something.

It actually sounds like a fun project.

Post reply on HN