Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

61–70 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#61

Post by thr0waway1239 »

pjc50 wrote: post #56
thr0waway1239 wrote: post #16

Earlier quoted context omitted.

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

> seems possible for a computer to detect content which is just mostly marketing But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

I agree. Its a pretty tough problem, however it is good to cross the bridge when it comes. If the search engine stays really niche, perhaps it may not even be worth it for the spammers, while doing enough to cater to the somewhat self-selecting audience. For example, the number of people who want to get to the front page of HN is likely to be a really minuscule fraction of people wanting to get to the top of search results.

Also, I wonder if is it possible to detect promotional content by analyzing things like call to actions and such?

Re: Show HN: Open-source search engine with 2bn-page index

#62

Post by petra »

dood wrote: post #60
dvirsky wrote: post #55

Earlier quoted context omitted.

There were a few attempts at that in the past, one being http://omgili.com/ that now seems to return pretty much garbage. BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch…

Heh, classifying by "genre" is exactly what I was thinking of doing. Had some debate with myself if I should start by focusing on training for shopping pages (product pages & product reviews) - because that might make some money; or start by training for forums - which I'd enjoy a lot more. Or build a more general system which would definitely never work and never get finished. Google actually let you filter by "disc…

Google removed it because they aim at the mass market.

Another perspective: people who find answers in forums are less likely to be interested in ads. And who knows, maybe making search shitty(in so many ways, not just formus), ad revenues rise ?

Re: Show HN: Open-source search engine with 2bn-page index

#63

Post by ytjohn »

ashitlerferad wrote: post #40

Another open source search engine: http://yacy.net/

Thanks, I was trying to remember that one. I think that for any new, non-profit, search engine to be viable, it has to be decentralized. deusu.com takes 2-3 months to crawl 2bn pages. Yacy claims to be at 1.4bn. I don't know how long it takes for that index to get refreshed, but it has 600 peer operators. Even if Yacy has a weaker indexing algorithm, I imagine that 600 peers, each crawling and contributing their own set of sites must be faster than a single deusu node.

Yacy is also quite a bit more resilient.

I will say that I don't buy Yacy's "no censoring" statement. If I was a bad actor, I could run yacy on a computer with false dns and false certificates, and yacy could index my fake content with official looking URLs.

Re: Show HN: Open-source search engine with 2bn-page index

#64

Post by Jaruzel »

tempestn wrote: post #29
Jaruzel wrote: post #18

Earlier quoted context omitted.

I always had an inkling for a Search Engine that ONLY indexed the root page of every domain. Not sure if I'm right about this, but it sure would sort the chaff from the wheat for general purpose queries.

Seems like that would just give you all those made-for-seo sites that tend to have second-rate content at best. ie, you search for 'best electric lawn mower', and you'll get bestelectriclawnmowers.com, 10bestelectricmowers.com, etc. Those sort of sites exist for every imaginable topic, and in my experience are rarely worth visiting. I would almost want the opposite. The best content on most topics I've found tends to…

So what we really want is a more granular search system i.e. 'only forums/blogs' 'not shopping sites' etc?

Re: Show HN: Open-source search engine with 2bn-page index

#66

Post by pmontra »

Written in Delphi. I might be wrong but I don't see many people downloading and working on it. 30 day free trial and then you have to pay for the development environment. IMHO it's a non starter for an open source project but if it's the only language the author is comfortable with, well that's OK.

Re: Show HN: Open-source search engine with 2bn-page index

#68

Post by pbhjpbhj »

thr0waway1239 wrote: post #16
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

It seems that you could just use Google's algorithms and modify the site trust metric using a front-page spam-score, whilst reducing the effect of link-juice from links with associated marketing keywords ("buy the doohickey on this link", or whatever).

Keeping marketing sites high in your SERPs would make you way more money on referrals though.

Re: Show HN: Open-source search engine with 2bn-page index

#69

Post by hyperpape »

pjc50 wrote: post #56
thr0waway1239 wrote: post #16

Earlier quoted context omitted.

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

> seems possible for a computer to detect content which is just mostly marketing But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

Interesting that if this were a parsing problem (i.e. https://news.ycombinator.com/item?id=12478538), folks would immediately suggest accepting known good output, instead of trying to blacklist specific problems. The analogue in search would be something that looks more like a directory than an internet wide search engine.

Of course what killed directories in the early web is that they had no hope of scaling.

Re: Show HN: Open-source search engine with 2bn-page index

#70

Post by pbhjpbhj »

pjc50 wrote: post #56
thr0waway1239 wrote: post #16

Earlier quoted context omitted.

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

> seems possible for a computer to detect content which is just mostly marketing But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

If they hide their sell links, shipping baskets, closing pages and such then they'll kill the sales though.

You can have paid news, but it's not doing anything if the mark can't buy the product afterwards because you had to remove all associations with selling to get the news to rank.

Post reply on HN