Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

41–50 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#41

Post by webtechgal »

deusu wrote: post #19
mstolpm wrote: post #17

In addition to the lack of removing porn and the ordering of the results not priorizing "quality" sources, some of the indexed site data is at least 4-6 months old and has heavily changed since the last crawl. I even got 404 errors. That makes it very hard to really find use in the project other than for academic interest.

A fresh recrawl is currently running. Should take about 2-3 months. Newly crawled data will gradually replace older data during that time.

Great work, congrats. :-)

Here is some input based on my experience building a similar project at my former company. (We did not quite get to 2B pages, but were close to ~300M):

For creating a really viable (alternative) search engine, the freshness of your index is going to be a fairly important factor. Now, obviously, re-crawling a massive index frequently/regularly is going to need/consume some huge amounts of bandwidth + CPU cycles. Here is how we had optimized the resource utilization:

Corresponding to each indexed URL, store a 'Last Crawled' time-stamp.

Corresponding to each indexed URL, also store a sort-of 'crawl-history' (If space is a constraint, don't store each version of the URL, store only the latest one). On each re-crawl, store two data fields: time-stamp and a boolean if the URL content has changed since last crawl. As more re-crawl cycles run, you will be able to calculate/predict the 'update frequency' of each URL. Then, prioritize the re-crawls based on the update frequency score (i.e. re-crawl those with higher scores more frequently and the others less frequently).

If you need any more help/input, let me know and I'll be happy to do what I can.

HTH and all the best moving forward.

Re: Show HN: Open-source search engine with 2bn-page index

#42

Post by [deleted] »

deusu wrote: post #19
mstolpm wrote: post #17

In addition to the lack of removing porn and the ordering of the results not priorizing "quality" sources, some of the indexed site data is at least 4-6 months old and has heavily changed since the last crawl. I even got 404 errors. That makes it very hard to really find use in the project other than for academic interest.

A fresh recrawl is currently running. Should take about 2-3 months. Newly crawled data will gradually replace older data during that time.

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#43

Post by [deleted] »

thr0waway1239 wrote: post #16
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#44

Post by runarb »

deusu wrote: post #11
laurent123456 wrote: post #9

They need to filter porn out of their search results (even for common queries like "hat", there's only porn) and perhaps be more resilient to SEO techniques since it looks like there's lot of spam on top results. Queries with common words such as "cat" return almost only irrelevant results. I'd really like to see that kind of project working as a good alternative to Google, but as it is it's not really usable.

I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in a search-result, then that result gets downranked.

If you want you can use the now defunct web search engine boitho.com list of adult words. It is avalibal at https://github.com/searchdaimon/adult-words .

We mostly filtered out porn by using a two word phrase method. There were a lot of edge cases because many potentially dirty concepts are made up of words that are not bad when used alone. For example a text can have both "girls" and "nude" in it without being vulgar, but if it has the phrase "nude girls" the chance for it being pornografic is much higher.

Re: Show HN: Open-source search engine with 2bn-page index

#45

Post by deusu »

kowdermeister wrote: post #36
deusu wrote: post #28

Earlier quoted context omitted.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

That's great news, thanks for the info. Sorry for sounding harsh, for being a side project this is impressive. Have you also published the ranking mechanism? That way people might contribute you to improve it.

It's all open-source. So, yes.

Re: Show HN: Open-source search engine with 2bn-page index

#46

Post by webtechgal »

deusu wrote: post #28
kowdermeister wrote: post #23

Earlier quoted context omitted.

I don't know why would people use it to be frank. Lot better alternatives exists. > it could be good for a particular use case Namely? > or for learning from. The author admitted in the github readme that the code quality is rather bad. I also don't see a link to the search index, the only valuable component of this project.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

It would be great if you can share (at least) some information about the kind of hosting setup you're using, how much of bandwidth and how long it took to crawl and index the 2B pages.

Re: Show HN: Open-source search engine with 2bn-page index

#47

Post by webtechgal »

webtechgal wrote: post #41
deusu wrote: post #19

Earlier quoted context omitted.

A fresh recrawl is currently running. Should take about 2-3 months. Newly crawled data will gradually replace older data during that time.

Great work, congrats. :-) Here is some input based on my experience building a similar project at my former company. (We did not quite get to 2B pages, but were close to ~300M): For creating a really viable (alternative) search engine, the freshness of your index is going to be a fairly important factor. Now, obviously, re-crawling a massive index frequently/regularly is going to need/consume some huge amounts of ban…

We had also (obviously) built a (proprietary) ranking algo that took into account some 60+ individual factors. If it can be of any help, I'll create a list and send it to you.

Re: Show HN: Open-source search engine with 2bn-page index

#48

Post by deusu »

webtechgal wrote: post #46
deusu wrote: post #28

Earlier quoted context omitted.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb. There is also a free API in beta-test right now. Will probably be ready for official release next week.

It would be great if you can share (at least) some information about the kind of hosting setup you're using, how much of bandwidth and how long it took to crawl and index the 2B pages.

4 servers in total.

2 are used for crawling, index-building and raw-data storage. Quadcore, 32gb RAM, 4tb HDD and 1gbit/s internet connection on each of these. They are rented and in a big data-center. Crawling uses "only" about 200-250mbit/s of bandwidth.

2 servers for webserver and queries. Quadcore, 32gb RAM. One with 2x512gb SSD, the other with only 1x512gb SSD. These servers are here at home. I have cable internet with 200mbit/s down, 20mbit/s up. Static IPs obviously.

A full crawl currently takes about 3 months.

Re: Show HN: Open-source search engine with 2bn-page index

#49

Post by ddorian43 »

webtechgal wrote: post #47
webtechgal wrote: post #41

Earlier quoted context omitted.

Great work, congrats. :-) Here is some input based on my experience building a similar project at my former company. (We did not quite get to 2B pages, but were close to ~300M): For creating a really viable (alternative) search engine, the freshness of your index is going to be a fairly important factor. Now, obviously, re-crawling a massive index frequently/regularly is going to need/consume some huge amounts of ban…

We had also (obviously) built a (proprietary) ranking algo that took into account some 60+ individual factors. If it can be of any help, I'll create a list and send it to you.

Why not write that list here ?
Post reply on HN