Show HN: Open-source search engine with 2bn-page index • Hacker News Board

Re: Show HN: Open-source search engine with 2bn-page index

#11

Post by deusu » Tue, Sep 13, 2016, 10:11 AM UTC

laurent123456 wrote: post #9

They need to filter porn out of their search results (even for common queries like "hat", there's only porn) and perhaps be more resilient to SEO techniques since it looks like there's lot of spam on top results. Queries with common words such as "cat" return almost only irrelevant results. I'd really like to see that kind of project working as a good alternative to Google, but as it is it's not really usable.

I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in a search-result, then that result gets downranked.

Re: Show HN: Open-source search engine with 2bn-page index

#12

Post by [deleted] » Tue, Sep 13, 2016, 10:12 AM UTC

kowdermeister wrote: post #8

diggan wrote: post #4

Earlier quoted context omitted.

Not saying that it's better but one of the main selling-points of DeuSu seems to be that it's fully open source and independent search index. Duckduckgo, if I remember correctly, is not 100% open source and get their search index from Yahoo (or maybe Bing, not sure)

If it's not good, the it doesn't matter if it's OSS or not.

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#13

Post by anewhnaccount » Tue, Sep 13, 2016, 10:12 AM UTC

kowdermeister wrote: post #8

diggan wrote: post #4

Earlier quoted context omitted.

Not saying that it's better but one of the main selling-points of DeuSu seems to be that it's fully open source and independent search index. Duckduckgo, if I remember correctly, is not 100% open source and get their search index from Yahoo (or maybe Bing, not sure)

If it's not good, the it doesn't matter if it's OSS or not.

Good for what? Even though this isn't good for use as an every day general purpose search engine, it could be good for a particular use case perhaps with some adaptation or for learning from.

Re: Show HN: Open-source search engine with 2bn-page index

#14

Post by scandox » Tue, Sep 13, 2016, 10:12 AM UTC

Every time I see new search engine projects I remember this: https://en.wikipedia.org/wiki/Cuil

I note that Dr Anna Patterson is back with Google. She wrote this in 2004: http://queue.acm.org/detail.cfm?id=988407

Re: Show HN: Open-source search engine with 2bn-page index

#15

Post by deusu » Tue, Sep 13, 2016, 10:13 AM UTC

NKCSS wrote: post #10

Fun, but overal quality seems a bit lacking. When I search myself; the top 10 results don't even have my last name ('Kusters') and just shows pages that have the word 'Nick'. I suppose you don't use a form of LSA to score the search results? Maybe it's too specific, but afaik mainstream search engines seem to give somewhat consistent results here. https://deusu.org/query?q=nick+kusters Looking at the code ( https://g…

In my experience this is usually caused by the fact that even 2bn pages aren't that many nowadays. The index needs to get bigger to better find (and rank) long-tail results like queries like this.

Re: Show HN: Open-source search engine with 2bn-page index

#16

Post by thr0waway1239 » Tue, Sep 13, 2016, 10:18 AM UTC

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results.

It would likely be really niche, but it could become the anti-Google, which would be great when people actually seek an alternative to all the noise.

I think it has to be a non-profit like Wikipedia itself, I cannot imagine a model where it can also make money. The submitted site is a candidate but it has to improve the search quality as other commenters pointed out.

Re: Show HN: Open-source search engine with 2bn-page index

#17

Post by mstolpm » Tue, Sep 13, 2016, 10:18 AM UTC

In addition to the lack of removing porn and the ordering of the results not priorizing "quality" sources, some of the indexed site data is at least 4-6 months old and has heavily changed since the last crawl. I even got 404 errors. That makes it very hard to really find use in the project other than for academic interest.

Re: Show HN: Open-source search engine with 2bn-page index

#18

Post by Jaruzel » Tue, Sep 13, 2016, 10:19 AM UTC

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I always had an inkling for a Search Engine that ONLY indexed the root page of every domain. Not sure if I'm right about this, but it sure would sort the chaff from the wheat for general purpose queries.

Re: Show HN: Open-source search engine with 2bn-page index

#19

Post by deusu » Tue, Sep 13, 2016, 10:21 AM UTC

mstolpm wrote: post #17

In addition to the lack of removing porn and the ordering of the results not priorizing "quality" sources, some of the indexed site data is at least 4-6 months old and has heavily changed since the last crawl. I even got 404 errors. That makes it very hard to really find use in the project other than for academic interest.

A fresh recrawl is currently running. Should take about 2-3 months. Newly crawled data will gradually replace older data during that time.

Re: Show HN: Open-source search engine with 2bn-page index

#20

Post by laurent123456 » Tue, Sep 13, 2016, 10:21 AM UTC

deusu wrote: post #11

laurent123456 wrote: post #9

They need to filter porn out of their search results (even for common queries like "hat", there's only porn) and perhaps be more resilient to SEO techniques since it looks like there's lot of spam on top results. Queries with common words such as "cat" return almost only irrelevant results. I'd really like to see that kind of project working as a good alternative to Google, but as it is it's not really usable.

I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in a search-result, then that result gets downranked.

Yes I guess filtering them out would at least make the website SFW, and it would make it easier to show it to people. The issue seems to happen mainly with common words (which results also appear to be polluted with heavily SEO-ed websites).

I've also searched for less generic things like "xperia z5" and the results looked good.