Making an open source search engine is asking for it to be spammed.
Every spammer will have a test harness for the SE stack to ensure their spam is ranked as highly as possible.
We can do better than DuckDuckGo
361–370 of 383 posts
Re: We can do better than DuckDuckGo
#362One thing I always wished for is if there were a way to use duckduckgo bang searches in my browser without sending them through DDG. But apparently it's harder to implement than it sounds.
FYI, searx implements the DDG bang searches (1). There is no autocompletion (for now), but at least you can all of them in one meta search engine. The syntax is slightly different: "!!g " instead of "!g ".
(1) actually based on https://github.com/jivesearch/jivesearch/tree/master/bangs
Re: We can do better than DuckDuckGo
#363Making an open source search engine is asking for it to be spammed. Every spammer will have a test harness for the SE stack to ensure their spam is ranked as highly as possible.
By that logic, result quality should be relatively stable, because it’s unlikely that one entity will discover an effective way to game the system.
If it can be made to work, it should continue to work.
Re: We can do better than DuckDuckGo
#364> The search results suck Do they really though, for normal people that is?. Some of my searches today below, can't remember the exact terms I used. Mix of DDG and Google. 1) Walt Whitman, I wanted a basic overview of his work to satisfy some idle curiosity. DDG gave me his wikipedia page. Bingo 2) EAN-13 check digit. First result wikipedia telling me how to calculate it. I see it is simple and I have a long list in…
Apart from that, it's nice that is has no commercial bias. For instance if I search for a thing that is both a real thing and a product, I get the real thing returned.
Still, for really niche topics if search needed there is no way around Google. On the other hand Google is not the only way to explore the web, let alone auto-complete a url...
Re: We can do better than DuckDuckGo
#365Am I the only person who just doesn't have problems with DDG search results? What am I doing wrong (or right), here? I put a thing in and find it. I just don't use Google any more. Genuinely curious why it's working for me and such garbage for everyone else.
I say about 50% I'm good with DDG. About 1/3 of the time I add !g, usually for weird error messages and tech stuff. Honestly we shouldn't be using Google for everything. Why not just search StackExchange or Github issues directly for known bug problems? If you need a movie, !imdb or !rt forward you to exactly where you want to really search on. If DDG or Google also included independent small blogs for movie results,…
Just the weekend I had the "weird error message" situation. DDG gave me 1 page with no relevant results while Google had no problem finding proper matches.
Would be nice if they could do better on queries like that... though funny thing is if they didn't respect privacy they probably could. Log any searches where a user looked for something and then tried the same thing prefixed with !g. Use those for figuring out where to focus efforts and what to test with.
Re: We can do better than DuckDuckGo
#366Check out the serious difficulties the Common Crawl had with crawling 1% of the public internet on donated money and then get back to me with a plan. This is really, really hard to do for free. Maybe talk to Gates :)
TFA specifically mentions not crawling the web, but using a curated list.
Re: We can do better than DuckDuckGo
#367The only place I've really found duckduckgo lacking is its meme game and maybe esoteric tech problems. Even with the techy stuff if I narrow my search to github and stackoverflow I usually find what I'm looking for.
Re: We can do better than DuckDuckGo
#368Earlier quoted context omitted.
Just use the string 'googlebot' in your user agent. After all, Google uses 'Mozilla' in their Google Bot user agent String for similar reasons - because sites might expect it.
https://developers.google.com/search/docs/advanced/verifying...
That's interesting. I know a few companies that don't run that check but instead just go off the user agent.
Re: We can do better than DuckDuckGo
#369Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…
No, you just want a faster horse.
Re: We can do better than DuckDuckGo
#370Earlier quoted context omitted.
Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identifying them is hard.
We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.