Live data from Hacker News

SeenBefore: A search engine for what you have seen before

seenbefore.com

81–90 of 97 posts

Re: SeenBefore: A search engine for what you have seen before

#81

Earlier quoted context omitted.

Wow, does 240 people even count as a sample. At Yahoo and Google log sizes its probably the error from cosmic rays in the data center.

If they selected them in a properly random way and had an effect close to 40% then yes, that probably does count as a sample.

As someone who signed up to coursera stats 101, err... Why 40%?

Re: SeenBefore: A search engine for what you have seen before

#82

Earlier quoted context omitted.

Storing a list of 1.7 million strings for us takes 70mb stored in memory. Testing for membership is an O(1) op. Very happy with it. We use mongo as a dumb data store as well as a bunch of other infrastructure tools, like http://circleci.com we could have only dreamt of years ago.

Why not use a bloom filter?

The main benefit of Bloom filters is that they can be made small. Given that his database takes only 70MB or so and he's not trying to ship this to devices that might have much in terms of space limitations, there would appear to be little point.

Re: SeenBefore: A search engine for what you have seen before

#83
post #75

Earlier quoted context omitted.

Why not use a bloom filter?

Maybe because of this fact (according to Wikipedia)?: "The more elements that are added to the set, the larger the probability of false positives."

That depends on its size, though. You can make it larger and get fewer false positives.

Re: SeenBefore: A search engine for what you have seen before

#84
post #82

Earlier quoted context omitted.

Why not use a bloom filter?

The main benefit of Bloom filters is that they can be made small . Given that his database takes only 70MB or so and he's not trying to ship this to devices that might have much in terms of space limitations, there would appear to be little point.

Eh, true, I guess redis is sufficiently awesome.

Re: SeenBefore: A search engine for what you have seen before

#85
post #28

Earlier quoted context omitted.

Porn sites are not recorded

How does your system define "porn sites"? What about if it was some porn site no one has ever heard of with an innocent-sounding name/domain?

He apprently uses a list of 1.7 million sites. But you can also blacklist sites and have any existing entries for it removed:

https://www.seenbefore.com/blacklist_items

Re: SeenBefore: A search engine for what you have seen before

#86
post #15

Earlier quoted context omitted.

And would it be possible to configure it to use my own "cloud"?

Definitely something we are looking into. Major barrier is the cost for someone keeping a server running 24*7 in cloud(Micro instance on AWS is 175 dollars a year).

Lots of us here have our servers.... Personally, being able to point it at one of my own servers and/or getting an API, would be fantastic.

Re: SeenBefore: A search engine for what you have seen before

#87
post #19

Interesting idea. Some quick questions: - How much data do you store per user? - How do I delete certain results? (preferably after the search comes back) - Another thing to consider is - After how much time does this just become as painful as finding that page through a search engine? - What version of the page gets stored? The latest or the one that I saw? I guess its one step better than Evernoting a page and addi…

Main issue with Evernoting and Bookmarking is that it requires an effort to say that today, this page is useful and I want to store it. Most pages I want to find are very things I did not think was useful at the time. Each unique page(unique as per the content) is stored per user. Our goal is to build the tools needed to find the information quickly, similiar to what hipmunk.com did for airline search. We have the ad…

The main thing I use bookmarks for is categorisation. If you add the ability to tag and/or add notes that becomes part of the search terms, that'd be the killer feature for me - I could throw out my 3500 bookmarks and remove Xmarks (at least if we could get a way of automatically getting our existing bookmarks installed).

I'm a paying Xmarks user, but if you were to add a way of tagging sites or adding a note, I'd happily pay for this instead. Just a freeform text field that I could add some keywords into that gets treated as part of the search would actually be sufficient for me.

Re: SeenBefore: A search engine for what you have seen before

#88

Co-founder here. This took us by surprise, we were planning to have Firefox and Safari support done by launch. At this stage, it is priceless to know if we are solving a real problem people have. Also, is this something people would pay for (loops back to if this is enough of a pain point). From the moment we start charging, is the moment we start learning.

See comment elsewhere: With tagging or (simple plain text) notes attached, absolutely. Even moreso with a simple API and/or support to push the cached content to my own server. If it could be selectively enabled for private content too, then even better (e.g. there's several extensive private Wiki's I use regularly that are not sensitive enough that I'd worry about getting them indexed, and I'd love to be able to tell you to index them but perhaps disable the caching).

Re: SeenBefore: A search engine for what you have seen before

#89

Hmm, this is similar to http://historio.us , which I built. However, this doesn't require any user interaction, which might work well. Do you store just the URL and depend on Google returning the results? How does it work exactly?

I remember thinking that http://historio.us was a neat idea.

But Seen Before requires less effort on my part as a user -> I am more likely to use it. I just continue to google as per normal and now I have an extra option on the right to filter results.

Re: SeenBefore: A search engine for what you have seen before

#90

Hmm, this is similar to http://historio.us , which I built. However, this doesn't require any user interaction, which might work well. Do you store just the URL and depend on Google returning the results? How does it work exactly?

I remember thinking that http://historio.us was a neat idea. But Seen Before requires less effort on my part as a user -> I am more likely to use it. I just continue to google as per normal and now I have an extra option on the right to filter results.

This is true. The use cases are a bit different, but I still don't know exactly how this works so I can't say.
Post reply on HN