Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

141–148 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#141

Post by Razengan »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

The "ideal" search engine is probably not possible without some sort of AI having access to all the content on internet.

Re: Show HN: Open-source search engine with 2bn-page index

#142

Post by deusu »

wila wrote: post #128
deusu wrote: post #28

Earlier quoted context omitted.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

Thank you for your work. Keep going on it. Wired you a small donation as I think it is important to have alternatives.

Thank you!

Re: Show HN: Open-source search engine with 2bn-page index

#143

Post by rspeer »

Razengan wrote: post #141
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

The "ideal" search engine is probably not possible without some sort of AI having access to all the content on internet.

> some sort of AI having access to all the content on internet

Isn't this a description of any decent search engine?

Re: Show HN: Open-source search engine with 2bn-page index

#145

Post by deusu »

vcool07 wrote: post #144

Any specific reason you've used pascal ? I thought that language got extinct long ago.

It's alive and well. The TIOBE index still lists it ahead of Ruby, Swift, Objective-C, GoLang...

And I started this software 20 years ago. Granted, a LOT of the software has changed since then. But I don't see a reason to throw away existing code unless it is in need of so much change that rewriting from scratch would be easier. And even then I might stick to what I know best, and what fits best with other parts of the software.

Re: Show HN: Open-source search engine with 2bn-page index

#146

Post by mack73 »

ganeshkrishnan wrote: post #135
mack73 wrote: post #123

Earlier quoted context omitted.

That is a lovely idea. Unfortunately, a scoring scheme has one foot in the indexing process (that thing that the google bot does) and another in the querying part, so switching schemes would often mean you would need to re-index your data to cater for the new metrics you now need for a new type of scoring.

Neither indexing nor querying does the ranking. Ranking is done after indexing and can be either tf-idf , pagerank or combination of that. Once the document similarity to the query is calculated, by for example vector space model, the documents are ranked by pagerank. What OP is saying that instead of pagerank we can have other ranking methods which is surely plausible.

Sure, but what I was saying was that what good is a new ranking method, when you only have at your disposal the same set of metrics as the method you are trying to replace? A new ranking would quite often mean adding new metrics. For example, when Lucene when from tf-idf to bm25 they added lots of new metrics to be able to cater for the new algorithm.

Re: Show HN: Open-source search engine with 2bn-page index

#147

Post by samstave »

allendoerfer wrote: post #138
samstave wrote: post #130

Earlier quoted context omitted.

Why not have a search engine with "sub-reddits" that can be subscribed to... Whereby - a site would self-identify as being in a particular genre, say "healthcare" - and I could launch a tab to the engine and set my sub to "health, health-tech, healthcare, medicine, etc.." and then do my search and only those sites that set their category will show up in that search - but if I dont find my search, I can then easily sl…

> Or has this been tried/found to be stupid? You are describing the keywords meta tag. While it is often told that competitors before Google did not use something like PageRank, which is not true, Google's PageRank algorithm was better and cheaper than the competitors' and effectively killed your idea 20 years ago.

Appreciate the insight....

But I find it slightly ironic that people are bitching about PageRank having slightly some issues with respect to the specificity of what they are searching for...

meaning that even though "killed this idea twenty years ago" we are coming back to the same problem...

Is that perhaps just due to the volume of info that is available on the web and the much more complex way we have categorized (mentally, not digitally) all the knowledge and information thats out there now?

Re: Show HN: Open-source search engine with 2bn-page index

#148

Post by ganeshkrishnan »

mack73 wrote: post #146
ganeshkrishnan wrote: post #135

Earlier quoted context omitted.

Neither indexing nor querying does the ranking. Ranking is done after indexing and can be either tf-idf , pagerank or combination of that. Once the document similarity to the query is calculated, by for example vector space model, the documents are ranked by pagerank. What OP is saying that instead of pagerank we can have other ranking methods which is surely plausible.

Sure, but what I was saying was that what good is a new ranking method, when you only have at your disposal the same set of metrics as the method you are trying to replace? A new ranking would quite often mean adding new metrics. For example, when Lucene when from tf-idf to bm25 they added lots of new metrics to be able to cater for the new algorithm.

did lucene go from tf-idf to okapi bm25? Surprising. Need to research it up.

We use tf-idf too but augment with page rank and clustering. gets more relevant docs

Post reply on HN