Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

121–130 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#121

Post by Keyframe »

austingulati wrote: post #116
Keyframe wrote: post #103

Earlier quoted context omitted.

I always had this, maybe stupid idea, of vouching referrals search engine. You pick a few sites you know are good and they (their 'webmasters') would vouch for new ones, and those could vouch for new ones, etc. Catch is, if one or two (whatever) of your child or grandchild vouched sites screws up, then they're toast, out of index, but so are you. That way you would pick wisely. Same idea would probably work for onlin…

This is very similar to PageRank except "vouch" is accomplished by linking to the other page.

Key difference being there's a responsibility of recommendations with consequences.

Re: Show HN: Open-source search engine with 2bn-page index

#122

Post by mack73 »

Jaruzel wrote: post #64
tempestn wrote: post #29

Earlier quoted context omitted.

Seems like that would just give you all those made-for-seo sites that tend to have second-rate content at best. ie, you search for 'best electric lawn mower', and you'll get bestelectriclawnmowers.com, 10bestelectricmowers.com, etc. Those sort of sites exist for every imaginable topic, and in my experience are rarely worth visiting. I would almost want the opposite. The best content on most topics I've found tends to…

So what we really want is a more granular search system i.e. 'only forums/blogs' 'not shopping sites' etc?

What we really want is a system that classifies your query as being one of "forum/blog/shopping" and then makes a scan only over that class of pages.

So on launch-day you would have checkboxes. On the one-year anniversary you´d have those checkboxes removed.

Google probably implemented this a couple of decades ago, though, so what we _really_ need is someone to come up with a new business model more attractive than Google's.

Ad-free is quite refreshing.

Re: Show HN: Open-source search engine with 2bn-page index

#123

Post by mack73 »

ma2rten wrote: post #51
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I would love to have a search engine that would allow you supply your own ranking function.

That is a lovely idea. Unfortunately, a scoring scheme has one foot in the indexing process (that thing that the google bot does) and another in the querying part, so switching schemes would often mean you would need to re-index your data to cater for the new metrics you now need for a new type of scoring.

Re: Show HN: Open-source search engine with 2bn-page index

#124

Post by GFischer »

pjc50 wrote: post #56
thr0waway1239 wrote: post #16

Earlier quoted context omitted.

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

> seems possible for a computer to detect content which is just mostly marketing But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

Extremely relevant XKCD:

https://xkcd.com/810/

But yes, even here on HN there are problems distinguishing between legitimate articles and paid news.

Re: Show HN: Open-source search engine with 2bn-page index

#127

Post by wila »

deusu wrote: post #115
rasengan wrote: post #112

Earlier quoted context omitted.

It would be a interesting data point to see how many of the 2b pages indexed are adult. lol.

I don't know. But I do know that the end-of-year statistics from search-engines about what people searched for, are complete BS. I have such a list for the German DeuSu page: https://deusu.de/blog/2015-12-03-alle_jahre_wieder_wonach_de... Warning! This is definitely NSFW! :)

LOL, seems that a big part of your users is searching for adult related things.

So if you filter out all the adult stuff you might make those users unhappy. Perhaps make it configurable?

Eg. add a checkbox for NSFW results or something better.

Re: Show HN: Open-source search engine with 2bn-page index

#128

Post by wila »

deusu wrote: post #28
kowdermeister wrote: post #23

Earlier quoted context omitted.

I don't know why would people use it to be frank. Lot better alternatives exists. > it could be good for a particular use case Namely? > or for learning from. The author admitted in the github readme that the code quality is rather bad. I also don't see a link to the search index, the only valuable component of this project.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

Thank you for your work. Keep going on it.

Wired you a small donation as I think it is important to have alternatives.

Re: Show HN: Open-source search engine with 2bn-page index

#129

Post by karussell »

deusu wrote: post #95
siegecraft wrote: post #94

Earlier quoted context omitted.

How does the search / indexing compare to sphinx or lucene?

I don't know sphinx at all, and my knowledge of lucene is very limited. Which means I don't know how they would compare to DeuSu.

You should really have a look into both. I'm curious: Why did you create a new software from scratch if you do not know existing?

If you'd base a project on lucene/solr/elasticsearch or sphinx you'd also increase the chance of contributing to it.

Re: Show HN: Open-source search engine with 2bn-page index

#130

Post by samstave »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Why not have a search engine with "sub-reddits" that can be subscribed to...

Whereby - a site would self-identify as being in a particular genre, say "healthcare" - and I could launch a tab to the engine and set my sub to "health, health-tech, healthcare, medicine, etc.." and then do my search and only those sites that set their category will show up in that search - but if I dont find my search, I can then easily slide out to other areas where I may not have thought what I was looking for would have identified with. Further - any post by any company/site could individually been given a topic to self-declare as... thus even if the company or site isnt necessarily in that space - their page or object could at least be a part of that result ranking....

Or has this been tried/found to be stupid?

Post reply on HN