Earlier quoted context omitted.
The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.
How would anyone block a crawler? A crawler is just a headless browser.
We can do better than DuckDuckGo
51–60 of 383 posts
Re: We can do better than DuckDuckGo
#52DDG does operate their own crawler[1], though they also do still rely on third parties[2]. [1] https://help.duckduckgo.com/duckduckgo-help-pages/results/du... [2] https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
Author didn't even DDG to find this out?
Re: We can do better than DuckDuckGo
#53My main problem with DDG is that there's no way to be sure they actually respect their users' privacy as they claim to. Ideally, services like theirs would be continuously audited by respectable, trusted organizations like the EFF.. multiple such organizations even. Then I'd have at least some reason to believe their claims of not collecting data about me. As it stands, I only have their word for it.. which in this d…
I don’t even care about the privacy. (Well, I do, but in this context I have no reasonable way to ensure it) What I do care about is trust-building and monopolistic practices. That, to me, is a great reason to use DDG instead of Google or even Bing.
I also prefer DDG's user interface over Google's. And DDG's !bang search shortcuts.
DDG has been my default search engine for years and its results are good enough for me 95% of the time. I only need to use Google as a fallback when searching for niche technical information or "needles in haystacks".
Re: We can do better than DuckDuckGo
#54Re: We can do better than DuckDuckGo
#55How would anybody ever know what the server is running and/or doing with the data you send it, regardless of if it is running open or closed source code? A service, running on somebody else's machine, is essntially closed. I think the only way to have an 'open' service is to have it managed like a co-op, where the users all have access to deployment logs or other such transparency. Even then, it requires implicit tru…
In theory, this is the kind of thing that the GPL v3 was trying to address: roughly speaking, if you host & run a service that is derived from GPL-v3'd software, you are obliged to publish your modifications.
But, I agree with you - and I don't think the author had really thought through what they were demanding, they made no mention of licensing other than singing happy praises of FOSS as if that would magically mean you could trust what a search engine was doing.
Re: We can do better than DuckDuckGo
#56> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…
But the email is already like this. It's the inbox providers who choose what domain is legit and new domains start from negative rating. Treating the web the same way doesn't sound too unnatural.
It would be bad if those in the positions profit by "authorizing" who is good though.
Re: We can do better than DuckDuckGo
#57The way I'd code a better search engine is I'd design an ML model that's trained to recognize handwritten HTML like this, and only add those to the index. It'd be cheap to crawl probably only needing a single computer to run the whole search engine. It'd resurrect The Old Web, that still exists, but just got buried beneath the spammy SEO optimized grifter web over the years as normies flooded the scene.
Re: We can do better than DuckDuckGo
#58Earlier quoted context omitted.
>How would you verify that for any centralized service, open source or not? Other centralized (search) services don't have their entire existence depending on this one factor. What is DDG if not alleged privacy? Just use Bing directly.
I don't understand that argument at all. What's the threat model? I think it's entirely reasonable to be in the following posture: I want as much privacy for my web searches as I can reasonably achieve without having to run a search engine myself. I'm willing to trust that search providers are not saving personally identifiable information or passively turning over search data to law enforcement if they claim that th…
I think we're on the same page. I was saying that if it were to be discovered that DDG lacks privacy then there would be no reason to use it over Bing since that is its raison d'etre.
>I'm willing to trust that search providers are not saving personally identifiable information or passively turning over search data to law enforcement if they claim that they are not in their terms of service.
Do other search companies disclose that they share data with the FBI, NSA, etc in their ToS? Genuinely don't know.
Re: We can do better than DuckDuckGo
#59One thing I always wished for is if there were a way to use duckduckgo bang searches in my browser without sending them through DDG. But apparently it's harder to implement than it sounds.
You absolutely can, at least in Firefox you can right click on search field, select "Add a Keyword for this search..." Then save it as bookmark and enter the keyword (you don't have to use !, but it is an option if you chose so).
You can also create such bookmark manually and use %s in the url as a placeholder where search query should be placed.
The manual configuration can be useful when there's no direct search field. For example freshports.org allows querying freebsd.org. I can add a bookmark with search keyword "fp" to point to https://freshports.org/%S
After that I can type in address bar: fp lang/python39 to land on https://freshports.org/lang/python39 (the capital %S doesn't escape special characters like /)
Re: We can do better than DuckDuckGo
#60SEO is crushing the utility of Google. It is pretty telling when you need to add things like site:reddit.com to get anything of value. Harnessing real user experiences (blogs, etc) is the key to a better search engine. This model unfortunately crumbles under walled gardens which is increasingly the preferred location of user activity.