Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

71–80 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#71

Post by beejiu »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

Another product (Discussions) that Google discontinued.

Re: Show HN: Open-source search engine with 2bn-page index

#72

Post by mox1 »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

Often times when searching for something, I would love this as well. I usually add phpbb, forums or discussion to the search keywords, but it's never 100%.

I would love something like this as well...

Re: Show HN: Open-source search engine with 2bn-page index

#74

Post by ganeshkrishnan »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

This search engine seems to use only tf-idf inverted index for it searches and then a vector space model for ranking the similarity.

A search for "java twitter bot" places more emphasis on "bot" then on Java and then on twitter which is what a tf-idf would do.

A good start like you said but it's miles away even from yahoo or bing.

Re: Show HN: Open-source search engine with 2bn-page index

#76

Post by adrianratnapala »

deusu wrote: post #28
kowdermeister wrote: post #23

Earlier quoted context omitted.

I don't know why would people use it to be frank. Lot better alternatives exists. > it could be good for a particular use case Namely? > or for learning from. The author admitted in the github readme that the code quality is rather bad. I also don't see a link to the search index, the only valuable component of this project.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

I think such publication is very important. I have no idea at all if open source search engines could ever work.

But if they can, I think a big part of it would be separation of the crawl index and the UI / prioritisation etc. Different people can work on those two ends of the problem and apply different philosophies.

Search only forums? Reject porn using XYZ method? Great! But they can all use the same database, or pick from the a common community of databases.

Re: Show HN: Open-source search engine with 2bn-page index

#77

Post by gkilmain »

tychuz wrote: post #73

And all javascript related questions still have w3schools as first result, god dammit.

I think for newbs who want to learn the fundamentals of web dev w3schools is a good resource. Even the people over at w3fools admit it. For a deeper dive though clearly MDN is the winner.

Re: Show HN: Open-source search engine with 2bn-page index

#78

Post by deusu »

skykooler wrote: post #75

It shows snippets of the web pages under each result; however, generally not the particular snippets that contain the search term. I would think that would be useful.

Yes, it would be better.

The snippets are currently the first 255 characters of the page's text. For snippets to be customized to the search term, I would have to store all the text of the page. And that would require a lot more disk space. Space that I can't afford at the moment.

Re: Show HN: Open-source search engine with 2bn-page index

#79

Post by deusu »

pmontra wrote: post #66

Written in Delphi. I might be wrong but I don't see many people downloading and working on it. 30 day free trial and then you have to pay for the development environment. IMHO it's a non starter for an open source project but if it's the only language the author is comfortable with, well that's OK.

Originally it was written in Delphi. But I now use FreePascal for the development. I'm even compiling both Windows and Linux versions on my Linux machine.

Re: Show HN: Open-source search engine with 2bn-page index

#80

Post by deusu »

ommunist wrote: post #67

DeuSu seems not indexing Cyrillic part of the Internet, and cannot give you insights for Greek, try https://deusu.org/query?q=ελιά . Is it Latin ANSI only index?

Only ASCII and German umlauts (äöüß) at the moment. The parser needs rewriting. It was originally written in pre-unicode times. :)
Post reply on HN