Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

31–40 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#31

Post by richmarr »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

The sheer scale required to attempt a new search engine is pretty staggering... it seems like one area where decentralisation might actually be worthwhile; the key obstacle being everyone's interest in gaming search results. I wonder if there's a useful application of ledgers that'd be useful in there somewhere...

Re: Show HN: Open-source search engine with 2bn-page index

#32

Post by micwo »

deusu wrote: post #25
micwo wrote: post #24

Deusu can't find deusu (or deusu.org) https://deusu.org/query?q=deusu

And why should it? You are already at the destination. No need to find it. :)

Try to find any other site by url:

https://deusu.org/query?q=news.ycombinator.com

Re: Show HN: Open-source search engine with 2bn-page index

#33

Post by gkst »

semi-extrinsic wrote: post #7
kowdermeister wrote: post #5

Strange, Wikipedia article is not on the first page and don't blame me for searching something non German thing :) https://deusu.org/query?q=berlin

It's pretty obvious that Google et al. do a lot of "custom" filtering like prioritising Wikipedia, removing porn from "obviously non-porn" searches etc. (That "Berlin" search gives porn as the 8th result.)

I doubt that Google prioritizes Wikipedia deliberately. Wikipedia has tons of backlinks, authority, trust, typically a high text to html ratio, probably a low bounce rate. Moreover, it is fast, works well on mobile and on and on. It's is just a very well done and useful site for users and search engines.

Re: Show HN: Open-source search engine with 2bn-page index

#34

Post by 0xmohit »

micwo wrote: post #32
deusu wrote: post #25

Earlier quoted context omitted.

And why should it? You are already at the destination. No need to find it. :)

Try to find any other site by url: https://deusu.org/query?q=news.ycombinator.com

Try to find `2 + 2 = 4`:

https://deusu.org/query?q=2+%2B+2+%3D+4

Even https://deusu.org/query?q=2+%2B+2+%3D+5 didn't yield any results. I was under the impression that it'd a message:

  2 + 2 = 5 for very large values of 2.

Re: Show HN: Open-source search engine with 2bn-page index

#36

Post by kowdermeister »

deusu wrote: post #28
kowdermeister wrote: post #23

Earlier quoted context omitted.

I don't know why would people use it to be frank. Lot better alternatives exists. > it could be good for a particular use case Namely? > or for learning from. The author admitted in the github readme that the code quality is rather bad. I also don't see a link to the search index, the only valuable component of this project.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

That's great news, thanks for the info. Sorry for sounding harsh, for being a side project this is impressive.

Have you also published the ranking mechanism? That way people might contribute you to improve it.

Re: Show HN: Open-source search engine with 2bn-page index

#37

Post by dood »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

Re: Show HN: Open-source search engine with 2bn-page index

#38

Post by kowdermeister »

gkst wrote: post #33
semi-extrinsic wrote: post #7

Earlier quoted context omitted.

It's pretty obvious that Google et al. do a lot of "custom" filtering like prioritising Wikipedia, removing porn from "obviously non-porn" searches etc. (That "Berlin" search gives porn as the 8th result.)

I doubt that Google prioritizes Wikipedia deliberately. Wikipedia has tons of backlinks, authority, trust, typically a high text to html ratio, probably a low bounce rate. Moreover, it is fast, works well on mobile and on and on. It's is just a very well done and useful site for users and search engines.

My thoughts as well. They don't need special treatment to be in the top 3.
Post reply on HN