Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

131–140 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#131

Post by ganeshkrishnan »

6502nerdface wrote: post #99
ganeshkrishnan wrote: post #74

Earlier quoted context omitted.

This search engine seems to use only tf-idf inverted index for it searches and then a vector space model for ranking the similarity. A search for "java twitter bot" places more emphasis on "bot" then on Java and then on twitter which is what a tf-idf would do. A good start like you said but it's miles away even from yahoo or bing.

Wow, the contrast between what this engine returns for that query and what google returns is amazing. Literally zero relevant links from the former and only relevant links from the latter. Search relevance is a serious high-science research problem, and it's going to be tough to compete with established players that have probably man-centuries' worth of proprietary research IP and some of the world's best scientists.

>Search relevance is a serious high-science research problem

We are working on this scenario at http://www.shoten.xyz using document clustering and apache spark graphx+ giraph.

Ain't been an easy task so far but we have made some headway

Re: Show HN: Open-source search engine with 2bn-page index

#132

Post by greglindahl »

jahewson wrote: post #107
Taek wrote: post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

If you're looking for an open source web crawl, commoncrawl.org has billions of pages.

... and Common Search made an index of the homepages, too.

Re: Show HN: Open-source search engine with 2bn-page index

#133

Post by ap22213 »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

Oh, Is this the search wish-list thread? Ok: I just want a search engine that only indexes the sites that would be of interest to people like me.

For most of my searches over the last year, Google has been so broken that it's almost unusable. At this point, to get any relevant results, I have to anticipate how Google will work, and then trial and error 10-15 times until I find what I'm looking for.

But, if I happen to be looking for incorrect information from 2005, Google works like a charm.

Re: Show HN: Open-source search engine with 2bn-page index

#134

Post by greglindahl »

rshm wrote: post #114

As of aug 16, common crawl has 1.73n pages. For the complimentary set of urls, if any benefit you can use their data dump as seed. If the metadata (such as last modified) size of your index is small enough to upload to aws, you can also reduce your re-crawl efforts when they have a fresh release.

It doesn't have to be small to donate to Common Crawl, they have a free S3 bucket.

Re: Show HN: Open-source search engine with 2bn-page index

#135

Post by ganeshkrishnan »

mack73 wrote: post #123
ma2rten wrote: post #51

Earlier quoted context omitted.

I would love to have a search engine that would allow you supply your own ranking function.

That is a lovely idea. Unfortunately, a scoring scheme has one foot in the indexing process (that thing that the google bot does) and another in the querying part, so switching schemes would often mean you would need to re-index your data to cater for the new metrics you now need for a new type of scoring.

Neither indexing nor querying does the ranking. Ranking is done after indexing and can be either tf-idf , pagerank or combination of that. Once the document similarity to the query is calculated, by for example vector space model, the documents are ranked by pagerank.

What OP is saying that instead of pagerank we can have other ranking methods which is surely plausible.

Re: Show HN: Open-source search engine with 2bn-page index

#136

Post by ganeshkrishnan »

amirouche wrote: post #108
ma2rten wrote: post #51

Earlier quoted context omitted.

I would love to have a search engine that would allow you supply your own ranking function.

How would this work? You could boost the query term for instance, like it possible to boost the column score in postgresql but that is all. Otherwise allowing user to provide their own ranking function (which is itself an art) would not be pratical performance wise. It should be noted that search engine interface, the search box is already a DSL for the underlying algorithm that support OR/AND and NOT.

Its not postgres or RDBMS for text search. Its usually bigdata. For example we use apache spark to query parquet files on HDFS

Re: Show HN: Open-source search engine with 2bn-page index

#137

Post by allendoerfer »

pbhjpbhj wrote: post #68
thr0waway1239 wrote: post #16

Earlier quoted context omitted.

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

It seems that you could just use Google's algorithms and modify the site trust metric using a front-page spam-score, whilst reducing the effect of link-juice from links with associated marketing keywords ("buy the doohickey on this link", or whatever). Keeping marketing sites high in your SERPs would make you way more money on referrals though.

Solving algorithmic tasks by just building a ML model of your competitor's algorithm seems like a funny way to start. I imagine this to be the way "programming" will stop being a thing in a few hundred years.

At the moment for web search it probably would not work, because I imagine from feature extraction to result there are several models involved to create intermediate results.

Re: Show HN: Open-source search engine with 2bn-page index

#138

Post by allendoerfer »

samstave wrote: post #130
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Why not have a search engine with "sub-reddits" that can be subscribed to... Whereby - a site would self-identify as being in a particular genre, say "healthcare" - and I could launch a tab to the engine and set my sub to "health, health-tech, healthcare, medicine, etc.." and then do my search and only those sites that set their category will show up in that search - but if I dont find my search, I can then easily sl…

> Or has this been tried/found to be stupid?

You are describing the keywords meta tag.

While it is often told that competitors before Google did not use something like PageRank, which is not true, Google's PageRank algorithm was better and cheaper than the competitors' and effectively killed your idea 20 years ago.

Re: Show HN: Open-source search engine with 2bn-page index

#139

Post by allendoerfer »

mi100hael wrote: post #119
Taek wrote: post #83

Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…

Interesting perspective. What about the other extreme, though? This circumstance has only really existed in the past 20 years or so, maybe less. Why not just revert some of your behaviors? - Delete your Facebook account. If you really need it to keep in touch with people across the country, at least delete the app from your phone and don't leave it open in a browser tab. - Don't place asinine Amazon orders just becau…

I am on your side. Except I need to contact some friends and - more importantly - customers, so I cannot entirely ditch everything. I cannot get rid of Facebook (Pages, API, relatives, even some customers), Skype (customers) and Whatsapp (friends). I am waiting for Whatsapp to become available on either Ubuntu Touch or Firefox OS. I would use these before buying a dumbphone and an MP3 player.

Re: Show HN: Open-source search engine with 2bn-page index

#140

Post by allendoerfer »

kowdermeister wrote: post #38
gkst wrote: post #33

Earlier quoted context omitted.

I doubt that Google prioritizes Wikipedia deliberately. Wikipedia has tons of backlinks, authority, trust, typically a high text to html ratio, probably a low bounce rate. Moreover, it is fast, works well on mobile and on and on. It's is just a very well done and useful site for users and search engines.

My thoughts as well. They don't need special treatment to be in the top 3.

Wikipedia being ranked high is even an indicator for SEOs that an affiliate niche is not very competitive.
Post reply on HN