It catches most dupes, just not on sites like the NYT that have so many different urls.
Hacker News "Submit" functionality needs a Digg-like duplicate alert
11–17 of 17 posts
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#12It catches most dupes, just not on sites like the NYT that have so many different urls.
Maybe you could check if the title of the page is the same as well, not as an automatic detection, but instead as a reason to ask "are you sure this isn't the same as foo". This might prevent most of the NYTimes dupes.
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#13Usually when I've submitted a dupe story, if it already exists it just adds a vote to the pre-existing story. Where the logic either breaks (or allows people to subvert it) is where different URL's can get you to the same story. Often times URL's contain some superfluous flags that don't change the content served, but just serve to log some referrer or layout type data (I'm sure everyone reading this site gets how th…
Human dupe-detection would be an excellent extension to this process.
Why not just compare the content at the other end of the link with the contents of existing links.
It wouldn't be that hard. Whenever a link is submitted, YC's server would visit the link, get the response, strip all html tags and white space from it, then hash whatever is left. It would then store this hash value with the link. Whenever a new story is submitted, it is likewise hashed and then a check is made for an existing link with the same hash value. If it exists, it's a dupe, if not, allow it.
This would be an extra check to the existing dupe URL string of course. It still wouldn't catch every single thing, but it should eliminate quite a few easy dupes.
If that turns to have a low success rate, try hashing the page title or maybe the http headers.
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#14Earlier quoted context omitted.
Human dupe-detection would be an excellent extension to this process.
Why not just compare the content at the other end of the link with the contents of existing links. It wouldn't be that hard. Whenever a link is submitted, YC's server would visit the link, get the response, strip all html tags and white space from it, then hash whatever is left. It would then store this hash value with the link. Whenever a new story is submitted, it is likewise hashed and then a check is made for an…
A single comment or timestamp would change the hash.
Maybe the , or the contents of the first or something would be a better proxy.
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#15Earlier quoted context omitted.
Why not just compare the content at the other end of the link with the contents of existing links. It wouldn't be that hard. Whenever a link is submitted, YC's server would visit the link, get the response, strip all html tags and white space from it, then hash whatever is left. It would then store this hash value with the link. Whenever a new story is submitted, it is likewise hashed and then a check is made for an…
A single comment or timestamp would change the hash. Maybe the , or the contents of the first or something would be a better proxy.
Yeah that is what I was thinking when I added that last line.
For some reason I initially wasn't thinking about comments... so the title would be a much better proxy.
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#16It catches most dupes, just not on sites like the NYT that have so many different urls.
Maybe you could check if the title of the page is the same as well, not as an automatic detection, but instead as a reason to ask "are you sure this isn't the same as foo". This might prevent most of the NYTimes dupes.
Would it be that hard to just take a fulltext index of each page that hits the hot page? From there, just show anything with some >N% similarity (probably 98 or so, as text ads can affect the site a little bit.)
Re: Hacker News "Submit" functionality needs a Digg-like duplicate alert
#17Earlier quoted context omitted.
Maybe you could check if the title of the page is the same as well, not as an automatic detection, but instead as a reason to ask "are you sure this isn't the same as foo". This might prevent most of the NYTimes dupes.
Would it be that hard to just take a fulltext index of each page that hits the hot page? From there, just show anything with some >N% similarity (probably 98 or so, as text ads can affect the site a little bit.)
With your comment, I saw for the first time how the semantic web might be useful.