Well, Cuil had a lot of money and couldn't do it. I don't know how you quantify your assertions but I suspect that if you brought back 2005 Google it would be easily gamed and struggle to deal with social media sites where a lot of content people are looking for is now found.
Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
401–410 of 492 posts
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#402Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
lmao, hopefully the C code isn't nearly as bad as your html
Please make your substantive points without snark or swipes. We ban accounts that do the latter, because it's poisonous to the culture we're trying to develop here.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#403Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
I just looked myself up in your search engine and I can confirm that it finds stuff old enough that google wouldn't find them (eg: and old patch I submitted on gnu savannah).
I tried looking up a game I'm interested in and the second results cluster from your search engine is a reddit thread about linux support for that game... I love this.
Great job!
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#404Earlier quoted context omitted.
Especially since Cloudflare went public back in 2019, at which point any investors cashed out. - Sincerely, a Google employee who has nothing to do with the investment branch of the company
> at which point any investors cashed out. Well, actually that is also not true. At IPO preferred stocks convert to common but the investors can keep their ownership, they can but don't have to cash out or can only partially cash out. Investors can also keep board seats in many (or most?) cases.
In this example, I don’t think it matters if Google Ventures kept their shares or not. So long as they are treated as any other stock holder, I don’t see an issue. If they still maintain a board seat, then there might be an issue, but I don’t see a problem with simply holding shares.
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#405Ongoing related thread:
Gigablast Search Engine - https://news.ycombinator.com/item?id=29421898 - Dec 2021 (10 comments)
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#406Earlier quoted context omitted.
You inspired me to try an even less specific search: thing Subjectively felt the gigablast results were a relative delight.
No bad idea. At the risk of being sidelined: "philosophy" was not so a bad term either. Start with an arbitrary Wikipedia link and click on the first keyword of the summary after the linguistic annotations (or other annotations in brackets) and repeat the process until you reach a loop. You will almost always end with "philosophy" -> "metaphysics" -> "philosophy" -> ... This works for "Berlin", "history" and "Caesar"…
that's tripped out. where did you hear about that?
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#407Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
I tried searching for an answer, but how do you get a site added to your directory? Who maintains it? Directories are a real PITA to maintain with any quality.
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#408Earlier quoted context omitted.
It's not quite that easy. Have you ever tried it? See my post below. Basically, yes, I've done it, but i had to go through a lot and was lucky enough to even get them to listen to me. I just happened to know the right person to get me through. So, super lucky there. Furthermore, they have an AI that takes you off the whitelist if it sees your bot 'misbehave', whatever that is. So if you have a certain kind of bug in…
Cloudflare is not the only gatekeeper, too. Keep that in mind. There's many others and, as an upstart search engine operator, it's quite overwhelming to have to deal with them all. Some of them have contempt for you when you approach them. I've had one gatekeeper actually list my bot as a bad actor in an example in some of their documentation. So, don't get me wrong, this is about gatekeepers in general, not just onl…
But your treatment one could say sites fronted by cloudflares are part of a closed web
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#409Earlier quoted context omitted.
It's flawed from the get go if reddit is the basis.
As much as I like to hate on reddit (I'm a permanently suspended user), not every sub there is trash. There are some great subs there on very specific niche topics.
Badge of honour I'd say. What was your transgression?
Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?
#410Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…
what kind of index is Gigablast using? traditional inverted index like Lucene or something more esoteric? I know Google and Bing both use weird data-structure like BitFunnel https://www.microsoft.com/en-us/research/publication/bitfunn...
100% custom.