Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

391–400 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#391

Post by gbmatt »

easton wrote: post #235
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

You can be whitelisted so Cloudflare doesn't slow you down (or block you): https://support.cloudflare.com/hc/en-us/articles/36003538743...

It's not quite that easy. Have you ever tried it? See my post below. Basically, yes, I've done it, but i had to go through a lot and was lucky enough to even get them to listen to me. I just happened to know the right person to get me through. So, super lucky there. Furthermore, they have an AI that takes you off the whitelist if it sees your bot 'misbehave', whatever that is. So if you have a certain kind of bug in your spider, or your bot 'misbehaves', whatever that means is anyone's guess, then you're going to get kicked off the list. So then what? You have to try to get on the whitelist again? They have Bing and Google on some special short lists so those guys don't have to sweat all these hurdles. Lastly, their UI and documentation is heavily centered around Google and Bing, so upstart search engines aren't getting the same treatment.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#392

Post by betwixthewires »

1vuio0pswjnm7 wrote: post #383
betwixthewires wrote: post #360

Earlier quoted context omitted.

Wow, do you happen to have published your utility so that other people can play with it?

The problem is that (1) I am a minimalist and dislike lots of "features" and (2) I prefer extremely simple HTML that targets the links browser. Most users are probably using graphical, Javascript- and CSS-enabled browsers so while this may work great for me, it may be of little interest to others who have higher aesthetic expectations. Another problem is I prefer to write tiny shell scripts and small programs in C th…

Well something like that would be interesting to a particular demographic. I prefer minimal aesthetic cruft as well, and like terminal stuff like links.

If you ever do decide to publish, be sure to post it here!

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#393

Post by kazinator »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Is there a way to get the results to be formatted for desktop?

It looks like the layout is hard-coded for a mobile browser, in portrait mode.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#394

Post by [deleted] »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

[deleted]

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#395

Post by gbmatt »

gbmatt wrote: post #391
easton wrote: post #235

Earlier quoted context omitted.

You can be whitelisted so Cloudflare doesn't slow you down (or block you): https://support.cloudflare.com/hc/en-us/articles/36003538743...

It's not quite that easy. Have you ever tried it? See my post below. Basically, yes, I've done it, but i had to go through a lot and was lucky enough to even get them to listen to me. I just happened to know the right person to get me through. So, super lucky there. Furthermore, they have an AI that takes you off the whitelist if it sees your bot 'misbehave', whatever that is. So if you have a certain kind of bug in…

Cloudflare is not the only gatekeeper, too. Keep that in mind. There's many others and, as an upstart search engine operator, it's quite overwhelming to have to deal with them all. Some of them have contempt for you when you approach them. I've had one gatekeeper actually list my bot as a bad actor in an example in some of their documentation. So, don't get me wrong, this is about gatekeepers in general, not just only Cloudflare and Cloudfront.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#396

Post by derekjdanserl »

KarlKemp wrote: post #363
RhysU wrote: post #231

Earlier quoted context omitted.

A subscriber-supported search engine sounds cool to me. Any precedent?

As a general rule, nobody is willing to pay what they are worth to advertisers. Facebook makes 70$ / y / user in the US. You would pay $70 for an ad-free Facebook? Congratulation, you must be an above-average earner. Also: your value to advertisers just tripled. If you are willing to pay $210, it will immediately triple again.

Great point! So simple, but as someone who has never worked on this side of things I never thought about it.

How would legal limitations on data collection, like GDPR, influence the ratio? None? Only an insignificant degree? Or enough to actually influence business decisions?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#397

Post by lowwave »

robbomacrae wrote: post #246
ampersandy wrote: post #243

Earlier quoted context omitted.

Requiring users to know what sites they want in advance somewhat defeats the purpose of a search engine, no?

Not at all. You only have to fail the first request. It is an approach I took with my own attempt at a search engine way back! In fact I know personally that there is at least one patent out there that suggests initial 1st time request users being asked to provide the appropriate response as an efficient way to teach systems for future users. Obviously failing first requests isn't ideal but for popular requests it qu…

> Obviously failing first requests isn't ideal but for popular requests it quickly becomes insignificant.

The first request can also be called asynchronously, and display a message to the user that it is 'processing....'.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#398

Post by kingcharles »

entropie wrote: post #307
gbmatt wrote: post #205

Earlier quoted context omitted.

there's some stuff here : https://github.com/gigablast/open-source-search-engine

Holy, thats a huge codebase. Github even shows no code/syntax hl for many cpp files because they are so big. I fiddled around and searched for some not so well known sites in germany and the results were surprisingly good. But it looks really... aged.

Holy shit. Click on random .cpp file. Browser hangs. O_O

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#399

Post by gbmatt »

Archelaos wrote: post #227
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

I tried out four search words with your search engine, and I am not convinced that it is mainly the index size and not the algorithm that is to blame for bad search results. There are way too much high ranking false positives. Here is what I tried: a) "Berlin": 1. The movie festival "Berlinale" 2. The Wikipedia entry about Berlin 3. Something about a venue "Little Berlin", but the link resolves to an online gaming si…

I'll admit I had not been working on the quality of single term queries as much as I should have lately. However, especially for such simple queries, having a database of link text (inbound hyperlinks and the associated hypertest) is very, very important. And you don't get the necessary corpus of link text if you have a small index. So in this particular case the index size is, indeed, quite likely a factor.

And thank you for the elaborate breakdown. It is quite useful and very informative, and was nice of you to present.

And I'm not saying that index size is the only obstacle here. I just feel it's the biggest single issue holding Gigablast's quality back. Certainly, there are other quality issues in the algorithm and you might have touched on some there.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#400

Post by Lhiw »

gbmatt wrote: post #395
gbmatt wrote: post #391

Earlier quoted context omitted.

It's not quite that easy. Have you ever tried it? See my post below. Basically, yes, I've done it, but i had to go through a lot and was lucky enough to even get them to listen to me. I just happened to know the right person to get me through. So, super lucky there. Furthermore, they have an AI that takes you off the whitelist if it sees your bot 'misbehave', whatever that is. So if you have a certain kind of bug in…

Cloudflare is not the only gatekeeper, too. Keep that in mind. There's many others and, as an upstart search engine operator, it's quite overwhelming to have to deal with them all. Some of them have contempt for you when you approach them. I've had one gatekeeper actually list my bot as a bad actor in an example in some of their documentation. So, don't get me wrong, this is about gatekeepers in general, not just onl…

I dunno if y'all realise this but I'd pay for a search engine that black holes CloudFlare and any other sites that think bots shouldn't read their sites.
Post reply on HN