Live data from Hacker News

Ask HN: Is there room for another search engine?

news.ycombinator.com

151–160 of 200 posts

Re: Ask HN: Is there room for another search engine?

#151

Earlier quoted context omitted.

Algolia is a game-changer. They made it so incredibly simple to add search to your website. I'm not talking about their widgets, I'm talking about their server-side integrations and their javascript client-side lib. It's like magic. See it in action here: https://stackshare.io/match

It's good in results, but too aggressive in updating live results -- the effect is laggy on systems I use, e.g., when searching HN.

That's the fault of the implementation client-side. You shouldn't fire off an ajax request on every key stroke but leave a certain timeout zone after keys are pressed to detect "yeah this guy has finish typing, let's search now".

Re: Ask HN: Is there room for another search engine?

#152
I beleive that there is a place in the world for a new search engine. People say that Google is good enough, but I find it hard to use Google to find the things that I search for. Here is how I would upon Google:

I often find that I search for "vegan pancake recipe" and end up at a page with lots of images and it is very hard for me to find the ingredients list. Google does a poor job here. They should give preference to simpler sites where it is easier for me to find the information I'm looking for. Instead, they seem to actually give preference to complex sites. If their job is to help me search for the information, then they shouldn't give links to haystacks. They have tried to improve upon this with their answer feature where they quote websites. This is, IMO, the wrong way to do things.

Instead, the search engine should be a desktop application which is more pervasive than a website can be. It needs to run natively, and not be cloud based, both for privacy, performance, and for the ability to integrate well with the system. When I search for "vegan pancake recipies", if the search engine is going to give me a result which contains 3-5 pages of text and images before the ingredients list, it should automatically scroll the web browser window down to the actual recipe.

This desktop application should also build a context profile based on what I am doing on my computer. This context profile shouldn't be uploaded to the internet, but it is still usefull. For example, I should be able to select a string in my terminal and press the search icon in the sytem tray. This should bring up a stack exchange question containing the exact text of the string I selected.

I should also be able to select a set of websites which I want to use as my search "domain". I might give my search domain as "the documentation to Python3, the Docker API reference, and stack exchange". This whould make it so that those "feeling lucky" links would work much better.

The search engine should also present image results which are NOT WATERMARKED before ones which are!

I should be able to write a markup for things I need to search for, and then enter a "search engine research wizard". The makup would look like so:

"We had a great time at [Park on that hill in prague???] park. It was so sunny! The temperature was [Prague temperature on 27th of march ???] which is [Average Prague temperatures in March ???] for this time of year."

The search engine would then, when shown this text, would allow you to right click on the bracketed areas, search for the text in them, and then, by selecting parts of wikipedia articles, fill in the blanks.

The search engine should use accessibility APIs to record the text of the windows that I have open. I should then be able to use the search engine as a kind of memory store which I can search. If I want to know what that awsome new tiling window manager written in Rust was called, I should be able to search full text of my browsing history and open up the previous HN page where the tiling window manger was presented.

Re: Ask HN: Is there room for another search engine?

#153
post #40

Earlier quoted context omitted.

7% for Bing? That's huge! (Also, how about DuckDuckGo? I've got the (admittedly gut) feeling that it should at least outperform Ask and Excite.)

DuckDuckGo doesn't crawl/index by itself--it partners with other companies to use their search indexing. Bing is one of their primary sources of indexing, actually, although Wikipedia tells me that DDG's indexing is a compilation of "about 50 sources". https://en.wikipedia.org/wiki/DuckDuckGo

Interesting, I had no idea. I wonder if they have or ever had plans to index themselves? The wiki link mentions they have do their own crawler.

Re: Ask HN: Is there room for another search engine?

#155
post #87

Earlier quoted context omitted.

Yes, but Elasticsearch requires a dedicated server, you may have to use a river, you may have to shard, etc. Some people (most people?) don't want to think about shards & rivers and just want something to work. Algolia seems to do that.

I know what a shard is, but what's a river? I've never heard anyone use that term in a technical context before.

In Elasticsearch a "river" is the description for the pushing of data from your primary store (A database more than likely) into the elasticsearch index.

ES actually started with Rivers being part of Elastic but since deprecated it. You will still find people talking about "rivers" though as a description for however they are updating the index.

Re: Ask HN: Is there room for another search engine?

#156
I think there is and it's glaringly obvious. "If you don't pay for a service, you are the merchandise." With Google that is obviously true. If you can build an engine where you can demonstrate that you are not sniffing on the user, you will be able to charge a user fee, and a lot of people will gladly pay for it.

Re: Ask HN: Is there room for another search engine?

#157
post #79

Earlier quoted context omitted.

Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in…

Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot: https://support.google.com/webmasters/answer/80553?hl=en

[deleted]
Post reply on HN