Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

201–210 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#201
We are building one [1] as well as a few other people that I am aware of with different approaches and business models.

We also need to be aware that when we remember past times it usually carries a romantic, nostalgic note. Web is very different than it was 15 years ago and the problem of search has evolved.

What you are looking for is basically 'grep for the web' but it is just one facet of search that we use today. 15 years ago you would not get an instant answer to a question like you do today and many users would not be able to live without that today. There are also maps and location based answers, all sorts of widgets like translation etc. Also world became more polarized so an objective best search result became more difficult to produce, specially for events covered in news, which means bias inevitably starts to creep in.

This is not to say that Google is good or bad today, it is what it is and they are doing best they can. Startups like ours see an opportunity on the market, in large part to help savvy users find what they want.

[1] https://kagi.com

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#202
post #185

I would use a search engine that only indexed Reddit, Stack Exchange, Wikipedia, and a small number of other sites. And that specifically blocked Pinterest, Quora, most non-personal “blogs”, etc. People suggest DDG ! operators, but I don’t want to use a site’s (bad, single-site) search box. I want a multi-site SERP that only displays results from known good sites, which are customizable.

If I could add sites I liked to the index that'd be great. Find a blogger/hacker I like? Add to the index. Can I share my index with others? Can I include their indices in my searches?

Search engine as a social media platform? If I follow you, now I can search in your indices?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#203

Earlier quoted context omitted.

For "Bicycle" it would suck but I don't often use search engines that way, for "High Timber ALX 29" you'd probably get something like this: https://www.schwinnbikes.com/products/high-timber-alx-29?var... I wouldn't use it for everything but sometimes that is the exact behavior that I want. I'd use duck duck go for more general searches.

That is the top hit on google for that search, so what’s your complaint?

Take a random part number off your car, or a portion of a error message and try finding that. It's annoying to have to scroll down over a page or two of autogenerated SEO answers to get to something useful. The first result to appear on the internet is less likely to be SEO and more likely to be the manufacturers documentation or the git commit that spawned your error. It isn't always, but that's why you have more then one search engine.

Secondarily I think a search engine that is very simple in it's model and operation is useful for more general free speech purposes. If the major search engines decide they don't like a site like the pirate bay a search for '"Pirate Bay" And "Torrents"' on a search engine that does not curate could still get you there. I guess the point is without curation you have to work harder to find what you want, but nobody is actively preventing you from finding anything. It would help keep everybody honest.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#205
post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Interesting. I had some interests in building a search engine myself (for playing around ofcourse). I had read a blog post by Michael Nielson [1] which had sparked my interest. Do you have any written material about your architecture and stuff like that? Would love to read up. [1]: https://michaelnielsen.org/ddi/how-to-crawl-a-quarter-billio...

there's some stuff here : https://github.com/gigablast/open-source-search-engine

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#206

Earlier quoted context omitted.

This. DDG is my primary search engine now and has been for awhile. I don't use Google anymore to search unless I really need to. The algos they use today are not the same classic ones that actually returned results.

And if you really need to, DDG !bangs[0] make a search as simple as "!g mother google help me". The keyword thing is also available in Firefox as a browser feature, and elsewhere I'm sure, but nevertheless, it makes switching to DDG easier. (Plus I can directly go to the wiki page by using "!w", "!gm" for google maps, etc.) [0] https://duckduckgo.com/bang

The only bang I use is !gvb since DDG doesn't support verbatim searches.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#207
post #205

Earlier quoted context omitted.

Interesting. I had some interests in building a search engine myself (for playing around ofcourse). I had read a blog post by Michael Nielson [1] which had sparked my interest. Do you have any written material about your architecture and stuff like that? Would love to read up. [1]: https://michaelnielsen.org/ddi/how-to-crawl-a-quarter-billio...

there's some stuff here : https://github.com/gigablast/open-source-search-engine

Thank you.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#208

I've been using kagi.com for a month or so now, and it consistently beats DDG and Ecosia for result quality. I'd guess it beats Google too, since last time I used Google it was nothing but ads and spam which is why I stopped.

Thank you for the vote of confidence! Better than Google is our goal, glad you perceive it that way.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#209
You might call this a search engine based on the principle of Information Neutrality.

“Information Neutrality is the principle to treat all information provided (by a service) equally. The information provided, after being processed by an information-neutral service, is the same for every user requesting it, independent of the user’s attributes, including, e.g., origin, history or personal preferences and independent of the financial or influential interest of the service provider, as well as independent of the timeliness of information."

I wrote about this in relation to search [0]. We need to be allowed more freedom to choose search engines and services. One (default or selected) choice for search is unhealthy. We shouldn't have to choose between Google or Bing; DuckDuckGo or Startpage; Brave or Ecosia; Mojeek or Gigablast ..... Personally I use all 8 of these and more, as also explained [0].

[0] https://blog.mojeek.com/2021/09/multiple-choice-in-search.ht...

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#210

Earlier quoted context omitted.

Is it not the processing of natural language?

Would you call addition a system of linear equations? No, you don't use the college senior label for the highschool freshman topic. You use the smallest label that fits. It's string processing. NLP is actually understanding the language. Stemming is simple string matching. Playing the technicality game to stretch fields to encompass everything you think even marginally related isn't being thorough or inclusive; it's…

> NLP is actually understanding the language.

That's actually not an accepted terminology. There's, indeed, this:

  https://en.wikipedia.org/wiki/Natural-language_understanding
Not sure why are you so adamant that yours is the "true meaning", when NLP existed long before machine learning and AI were used for it. And even if not, every term can be defined differently, so it should be normal to have different institutions/people define NLP differently.
Post reply on HN