Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

91–100 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#91

Post by [deleted] »

deusu wrote: post #84
Taek wrote: post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation. And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format. BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#92

Post by deusu »

jbb555 wrote: post #85

I think projects like this are really important because they help reduce the impression that big server projects are only meant to be done by big companies. The internet is becoming a content consumption medium for many people. I'm not sure I'll use this, but I'll try to... it all depends on how good it is. But I approve of the project so I sent a (very) small bitcoin donation to hopefully help fund it for a few more…

Thank you!

Depending on who you are (there were 2 bitcoin donations today), you funded either about 18 or 28 hours of operations. :)

Re: Show HN: Open-source search engine with 2bn-page index

#93

Post by siegecraft »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

I thought about doing the same thing, found http://boardreader.com/ and then moved on. No idea how useful it is.

Re: Show HN: Open-source search engine with 2bn-page index

#94

Post by siegecraft »

deusu wrote: post #84
Taek wrote: post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation. And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format. BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

How does the search / indexing compare to sphinx or lucene?

Re: Show HN: Open-source search engine with 2bn-page index

#95

Post by deusu »

siegecraft wrote: post #94
deusu wrote: post #84

Earlier quoted context omitted.

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on the documentation. And the crawl- and index-data will be available for download in a few weeks. It's also just a matter of documenting the data-format. BTW: I disagree with your points about privacy. I see DeuSu as a way of fighting back.

How does the search / indexing compare to sphinx or lucene?

I don't know sphinx at all, and my knowledge of lucene is very limited. Which means I don't know how they would compare to DeuSu.

Re: Show HN: Open-source search engine with 2bn-page index

#96

Post by webtechgal »

ddorian43 wrote: post #49
webtechgal wrote: post #47

Earlier quoted context omitted.

We had also (obviously) built a (proprietary) ranking algo that took into account some 60+ individual factors. If it can be of any help, I'll create a list and send it to you.

Why not write that list here ?

Good idea. However, I'll need to really exercise the gray cells to put together the list so it might take me a couple of days. Once done, I'll post it here.

Re: Show HN: Open-source search engine with 2bn-page index

#98

Post by webtechgal »

deusu wrote: post #57
supersan wrote: post #52

Hi, I find the Blog more interesting right now since I hope to find write-ups about how you were able to manage such a herculean task on your own? Crawling 2bn pages could take forever and could generate a huge bandwidth bills, so any lessons you learnt, pitfalls you faced, etc would be a great read.

Some issues that appeared over the years: Block outgoing connects to local IP nets in your firewall. Otherwise your hosting provider might think you are trying to hack them. Apparently there are a lot of links out there that point to hosts which resolve to private IP ranges. Another problem with following links is that you are bound to run across some that are malware command & control servers. Had several complaints…

> Run your own DNS server and even cache the hostname/IP pairs yourself.

This[1] might be a useful resource to get started:

[1] https://scans.io/

(Register and download the IPv4 Address Space data file to use as an initial cache and then append/update as you go.)

Re: Show HN: Open-source search engine with 2bn-page index

#99

Post by 6502nerdface »

ganeshkrishnan wrote: post #74
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

This search engine seems to use only tf-idf inverted index for it searches and then a vector space model for ranking the similarity. A search for "java twitter bot" places more emphasis on "bot" then on Java and then on twitter which is what a tf-idf would do. A good start like you said but it's miles away even from yahoo or bing.

Wow, the contrast between what this engine returns for that query and what google returns is amazing. Literally zero relevant links from the former and only relevant links from the latter. Search relevance is a serious high-science research problem, and it's going to be tough to compete with established players that have probably man-centuries' worth of proprietary research IP and some of the world's best scientists.

Re: Show HN: Open-source search engine with 2bn-page index

#100

Post by deusu »

laurent123456 wrote: post #20
deusu wrote: post #11

Earlier quoted context omitted.

I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in a search-result, then that result gets downranked.

Yes I guess filtering them out would at least make the website SFW, and it would make it easier to show it to people. The issue seems to happen mainly with common words (which results also appear to be polluted with heavily SEO-ed websites). I've also searched for less generic things like "xperia z5" and the results looked good.

I have the filter implemented now. It's not perfect yet, but it already filters out a lot of the NSFW stuff. Unless you explicitly search for it.

I'm gonna further improve this over the next days. Right now it's just a quick'n dirty hack. :)

Post reply on HN