101–110 of 110 posts
Earlier quoted context omitted.
Why are so many search engines trying to piggy back off other search engines? Can they not crawl sites themselves and cut out the extra layer? In a sea of bad websites I feel like if you started with a whitelist approach it could be a good basis.
Crawling is difficult these days as recaptcha and other measures keep bots out if you are not google or MS
Earlier quoted context omitted.
I think DDG is more of a legitimate competitor in the search space than other engines running off the Bing index. My understanding is that they've been pretty transparent about using Bing and Yandex as sources for its product, have done quite a bit of work to tailor the results they spit back, and maintain their own knowledge graph for features like instant answers. They seem to function more as a metasearch engine.…
Why are so many search engines trying to piggy back off other search engines? Can they not crawl sites themselves and cut out the extra layer? In a sea of bad websites I feel like if you started with a whitelist approach it could be a good basis.
Earlier quoted context omitted.
Qwant is a completely independent search engine though, as far as I'm aware it's not using Bing underneath the hood anymore
Where are you getting that from?
But reading the French version of Wikipedia (I happen to be French) and the conclusion is much more tenuous. So I guess I'm wrong!
Earlier quoted context omitted.
Crawling is difficult these days as recaptcha and other measures keep bots out if you are not google or MS
That is quite unfortunate. How the hell are new original search engines supposed to compete?
Earlier quoted context omitted.
That is quite unfortunate. How the hell are new original search engines supposed to compete?
I’ve worked in this space for over two years. They can’t. It’s simply not possible. The only feature that’s feasible to offer is privacy. You’ll never be able to catch up to Google or Bing with crawling.
I've been reseasrching some ideas.
Earlier quoted context omitted.
Speaking as someone who ran operations in a search company that had an api for accessing their search results, that api will get slammed. But not in a good way. Crooks use search engines to find pages that have exploitable js code or sql injection vulnerabilities. They use them to find unprotected comment sections so they can inject spam into them. They use them to build dossiers on people by scraping public informat…
Can't this be remedied by introducing a 1 second cooloff period per IP address?
Lots of botnets were highlighted this way, when one address searches for 'joomla v2.3', and the next IP searches for 'joomla v2.3', page=2, and the next IP searches for 'joomla v2.3', page=3, etc.
It was annoying but an interesting problem. We could implement any arbitrary policy and then watch as the bots adjusted to come in just at that policy limit. We banned entire Ukrainian ISPs (they were a big source at the time) and have VPN providers become the big users. We put in limits per day, I tried a "thermal" system where IPs gained "heat" by queries and "cooled" by idle time. We built a server with a "broken" IP stack that we could send the initial TCP connect to, the server would accept the connection and then never respond. A "black hole" if you will. The trick was we didn't actually keep[ sockets open we just pretended like we it was the other end of a TCP connection. It did everything correctly except complete the connection. That would cause any client using off the shelf IP stacks to hang indefinitely.
It is a game with no ending as one might say.
I'm not sure there is a way to create a search engine that can verifiably respect your privacy unless you were to self-host the entire thing. This one's hosted in France. France is part of Nine Eyes, among other things.
Private.sh encrypts queries on the client side and routes requests thru proxies to maximize privacy.
Earlier quoted context omitted.
Private.sh encrypts queries on the client side and routes requests thru proxies to maximize privacy.
Looks like an interesting project! It would still need serious audits since it's all browser-based (arbitrary JS could be added at any time and snoop on the results page) and there's a black box element at the point of retrieving results: we're hoping that the search provider is only sending encrypted results back, but I don't see how we have a guarantee that this is the case and that the query and results aren't bei…