Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can do better than DuckDuckGo

drewdevault.com

Post reply on HN
Print view View original

281–290 of 383 posts

Re: We can do better than DuckDuckGo

#281

Post by 867-5309 »

jbay808 wrote: post #247
867-5309 wrote: post #224

Earlier quoted context omitted.

any type of profiling, opt-in or not, may be used to identify users

I mean, hacker news can probably also identify users based on which articles they click on, and how often they jump straight to the comments. I hope they don't. But a system such as I'm describing is probably the only one that can be entirely consistent with the two disparate requirements of fully anonymizing users, and being useful to both programmers and ophiologists studying different things called "python".

what you are describing is relevance based on user input (be that cookies, search history, interests, a preference for x over y) that may be used as identifying information, which vastly de-anonymises the service. if a search query is too ambiguous then it can be refined. if the user knows they want a programming language and not a snake, they can let the search engine know themselves. don't sacrifice their anonymity for perceived usefulness

Re: We can do better than DuckDuckGo

#283

Post by flobosg »

vibrant_mclean wrote: post #282

> Crawling and indexing non-HTTP data sources (Gemini? Man pages? Linux distribution repositories?) > ⇒ This article is also available on gemini. Does anyone know what is gemini? I couldn't find relevant results in any of the search engines.

https://drewdevault.com/gemini.html

Re: We can do better than DuckDuckGo

#284

Post by 3np »

jimnotgym wrote: post #207

> The search results suck Do they really though, for normal people that is?. Some of my searches today below, can't remember the exact terms I used. Mix of DDG and Google. 1) Walt Whitman, I wanted a basic overview of his work to satisfy some idle curiosity. DDG gave me his wikipedia page. Bingo 2) EAN-13 check digit. First result wikipedia telling me how to calculate it. I see it is simple and I have a long list in…

They do. Something fundamentally changed at some point during the past couple of years. It used to be that DDG was the best for verbatim search (meaning I want to only have results were the exact words I search for are included).

Now, even with quotes, I routinely get a whole first page of results where my terms are not included anywhere. Google generally respect the quotes.

Re: We can do better than DuckDuckGo

#285

Post by 3np »

kringdom89 wrote: post #222
jimnotgym wrote: post #207

> The search results suck Do they really though, for normal people that is?. Some of my searches today below, can't remember the exact terms I used. Mix of DDG and Google. 1) Walt Whitman, I wanted a basic overview of his work to satisfy some idle curiosity. DDG gave me his wikipedia page. Bingo 2) EAN-13 check digit. First result wikipedia telling me how to calculate it. I see it is simple and I have a long list in…

I suspect that anyone who claims that Duckduckgo "Just works" only do english search. I usually do "english" / "mother tongue" searchs all day. Everytime, I need to remember to toggle the regional button otherwise I get attrocious results. Whereas google simply understand that if I'm searching using the english language it should prioritize english results while if I'm searching in another language it should prioriti…

I also find that even with the regional toggle off, my results are still skewed towards my location or the native language of it. This is true for both DDG and Google. I want results completely agnostic of where my IP happens to be positioned.

Re: We can do better than DuckDuckGo

#286

Post by lambda_obrien »

phire wrote: post #249
lambda_obrien wrote: post #22

Why couldn't several coordinating specialized search engines share their data via something like "charge the downloader" S3 buckets? Then you get an org like StackExchange who could provide indexed data from their site and the algorithms to search the data the most efficiently, GitHub can do the same for their specific zone of speciality, Amazon, etc. Then anyone who wants to use the data can either copy it to their…

I think your idea has merit. Though it would require development of the "charge to download" S3 buckets and infrastructure to support payments. There is also an economic issue where you have to calculate the download cost to also cover storage costs.

I think you can already set an S3 bucket to charge the downloader for bandwidth, that's what I was talking about, the storage part is harder. The storage costs could be borne via some sharing agreements between the commercial interests and the pay customers where the infrastructure could be provided for smaller indexers (storage, compute) and in exchange the provider can use the data freely for their own services. Or, we could have a micropayments system that aggregates towards the end of the month, something open source and free to use, maybe a blockchain, this is actually one place that tokens on a blockchain could work as a semi-decentralized payment system between the parties in agreement in my "vision" or whatever you want to call it.

Re: We can do better than DuckDuckGo

#289

Post by nessex »

kringdom89 wrote: post #222
jimnotgym wrote: post #207

> The search results suck Do they really though, for normal people that is?. Some of my searches today below, can't remember the exact terms I used. Mix of DDG and Google. 1) Walt Whitman, I wanted a basic overview of his work to satisfy some idle curiosity. DDG gave me his wikipedia page. Bingo 2) EAN-13 check digit. First result wikipedia telling me how to calculate it. I see it is simple and I have a long list in…

I suspect that anyone who claims that Duckduckgo "Just works" only do english search. I usually do "english" / "mother tongue" searchs all day. Everytime, I need to remember to toggle the regional button otherwise I get attrocious results. Whereas google simply understand that if I'm searching using the english language it should prioritize english results while if I'm searching in another language it should prioriti…

I really wish Google would prioritize English results for English searches consistently. I'm living in Japan as a native English speaker, and have my OS, browser and logged in Google account all configured for English only. Despite that, Google search results always prioritize Japanese language content. Every now and then (though not consistently) it gives me a yellow popup asking if I'd like English results instead, which is a bit disappointing given they already have all the information they should need to make a judgement call about that. Maybe the individual experience here depends on the languages and regions involved.

Re: We can do better than DuckDuckGo

#290

Post by Shared404 »

edgyquant wrote: post #280
Shared404 wrote: post #205

Earlier quoted context omitted.

> why build a search engine at all in this case? On a public scale, you could make an argument for tighter integration/better privacy with the lists. For example: Browser -----Request-to-SE-----> Search Engine ^ | | Unfiltered Results (In YAML/JSON) | | | V |--Desired Results------ Local Filtering/Rendering On a private scale, if you are only crawling sites on the allow list than you have the possibility of being abl…

Aside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source solution. Which is the biggest problem with duck duck go.

Do you have thoughts on implementing the distributed search?

I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.

Post reply on HN