Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source search engine with 2bn-page index

deusu.org

Post reply on HN
Print view View original

51–60 of 148 posts

Re: Show HN: Open-source search engine with 2bn-page index

#51

Post by ma2rten »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I would love to have a search engine that would allow you supply your own ranking function.

Re: Show HN: Open-source search engine with 2bn-page index

#52

Post by supersan »

Hi, I find the Blog more interesting right now since I hope to find write-ups about how you were able to manage such a herculean task on your own?

Crawling 2bn pages could take forever and could generate a huge bandwidth bills, so any lessons you learnt, pitfalls you faced, etc would be a great read.

Re: Show HN: Open-source search engine with 2bn-page index

#53

Post by [deleted] »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

[deleted]

Re: Show HN: Open-source search engine with 2bn-page index

#54

Post by Cyph0n »

DanBC wrote: post #27
swiley wrote: post #22

The site's interface is just incredibly pleasant compared to Google.com. I really hope the author sticks with it. Unfortunately I'm not sure it's usable right now, searching "group theory Wikipedia" never brings up a Wikipedia page (although maybe I should just be directly searching Wikipedia if that's what I wanted).

DuckDuckGo's approach of !bang searches, making duckduckgo the place[1] I go when I want to search another site, is really useful. [1] It's my default search engine in Chrome, so I use bang searching in the address bar.

Same here. The problem is that I find myself using `!g` way too often... I guess I'm not used to the DDG results page.

Re: Show HN: Open-source search engine with 2bn-page index

#55

Post by dvirsky »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

There were a few attempts at that in the past, one being http://omgili.com/ that now seems to return pretty much garbage.

BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch to train, and my classifier's algorithm was pretty crappy, but it showed some potential.

I think today modern search engine do that behind the scene, and try to diversify the result to include pages from multiple genres, but they usually don't let you choose.

Re: Show HN: Open-source search engine with 2bn-page index

#56

Post by pjc50 »

thr0waway1239 wrote: post #16
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

Speaking of which, it seems possible for a computer to detect content which is just mostly marketing, versus content which is not (based on how spam filters work). The search engine should just show a "marketing index" score right next to the result. Even better is to whitelist certain sites (Wikipedia,popular .edu and .org domains) to begin with and prioritize those results. It would likely be really niche, but it c…

> seems possible for a computer to detect content which is just mostly marketing

But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

Re: Show HN: Open-source search engine with 2bn-page index

#57

Post by deusu »

supersan wrote: post #52

Hi, I find the Blog more interesting right now since I hope to find write-ups about how you were able to manage such a herculean task on your own? Crawling 2bn pages could take forever and could generate a huge bandwidth bills, so any lessons you learnt, pitfalls you faced, etc would be a great read.

Some issues that appeared over the years:

Block outgoing connects to local IP nets in your firewall. Otherwise your hosting provider might think you are trying to hack them. Apparently there are a lot of links out there that point to hosts which resolve to private IP ranges.

Another problem with following links is that you are bound to run across some that are malware command & control servers. Had several complaints to my ISP after authorities took over control of one and used the C&C server's domain as a honeypot. My crawler is on a whitelist now.

I had one person who vehemently complained that I was trying to hack him, because the software downloaded his robots.txt. I'm NOT kidding! :)

Make sure your robots.txt parsing is working correctly. I had an undiscovered bug in the software at some time which basically caused it to think everything is allowed. Luckily someone was nice enough to let me know. And he was really nice about it. And he would have had every right to be angry.

A major bottleneck is DNS queries. Run your own DNS server and even cache the hostname/IP pairs yourself. Do not even think about using your IPS's DNS server. If you bombard them with 100+ DNS requests/s then they WILL be angry. :)

Re: Show HN: Open-source search engine with 2bn-page index

#58

Post by iplaw »

throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I've started appending "forum" to many of my queries when I am looking for answers from users.

Re: Show HN: Open-source search engine with 2bn-page index

#59

Post by MasterScrat »

dood wrote: post #37
throwaway13337 wrote: post #6

Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

Probably the easiest way would be to use Google Search APIs and do custom queries and/or filtering.

Re: Show HN: Open-source search engine with 2bn-page index

#60

Post by dood »

dvirsky wrote: post #55
dood wrote: post #37

Earlier quoted context omitted.

I'd love a search engine which only indexes forums. Something I've been thinking of doing for years, but it'd be a lot of work.

There were a few attempts at that in the past, one being http://omgili.com/ that now seems to return pretty much garbage. BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch…

Heh, classifying by "genre" is exactly what I was thinking of doing.

Had some debate with myself if I should start by focusing on training for shopping pages (product pages & product reviews) - because that might make some money; or start by training for forums - which I'd enjoy a lot more. Or build a more general system which would definitely never work and never get finished.

Google actually let you filter by "discussions" until a few years ago, so they certainly do this kind of classification. It didn't work perfectly but sometimes did the trick. Don't know why they removed that feature.

Post reply on HN