Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can do better than DuckDuckGo

drewdevault.com

Post reply on HN
Print view View original

121–130 of 383 posts

Re: We can do better than DuckDuckGo

#121

Post by ur-whale »

kilroy123 wrote: post #92
dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.

>I welcome any competition at this point.

Microsoft tried and failed to build a competitor and it's not like they have shallow pockets.

They grossly underestimated a number of aspects:

   - The huge number of man-years invested in Google's search quality stack hand-tuning and what it would take to replicate it.

   - The fact that the machine learning field was simply not ready to tackle the search quality problem.

   - The infrastructure required to build a crawler / indexer stack as good as Google's
I think in 2020, the second problem is within reach of many companies technically. It's mostly a matter of throwing enough money at optimized infrastructure.

However, replicating the search quality stack is going to be very hard, unless someone makes a huge breakthrough in machine learning / language modeling / language understanding at a thousandth of the cost it currently takes to run something like GPT-3.

The most likely candidate to execute properly on that last bit is - unfortunately - Google.

Re: We can do better than DuckDuckGo

#122

Post by wenbin »

It's almost impossible to build a decent web search engine from scratch today (i.e., build your own index, fight SEO spams, tweak search result relevance...). The web is already so big and so complex. Otherwise Google won't need to hire so many people to work on search alone.

If you didn't start at the very early stage of tiny web (e.g., Google in 1996 as a research project) and grew with the web over the past 20+ years, or you don't have super deep pocket (e.g., Microsoft Bing in mid 2000s), then it's almost impossible to build a decent web search engine within a few years.

It's possible to build vertical search engines on far smaller scale, far less complex, far less lucrative things that Google/Microsoft has little interest today (e.g., recipes [2], podcasts [3], gifs [4]...)

It's also possible to come up with a different discovery mechanism for web (or a small portion of web), other than a traditional complete web search engine. Essentially you don't cross moat to attack a huge castle (e.g., Google). Instead, you bypass the castle [1], as it becomes irrelevant.

[1] https://twitter.com/benedictevans/status/1038538688232226817...

[2] https://www.yummly.com/

[3] https://www.listennotes.com/

[4] https://giphy.com/

Re: We can do better than DuckDuckGo

#123

Post by 867-5309 »

jbay808 wrote: post #34
bscphil wrote: post #21

> they’ve demonstrated gross incompetence in privacy Not sure I buy the example that is given here. 1. It's an issue in their browser app, not their search service. 2. It's not completely indefensible: it allows fetching favicons (potentially) much faster, since they're cached, and they promise that the favicon service is 100% anonymous anyway. 3. They responded to user feedback and switched to fetching favicons loca…

Maybe instead of hard-coding these preferences in the search engine, or having it try to guess for you based on your search history, you can opt-in to download and apply such lists of ranking modifiers to your user profile. Those lists would be maintained by 3rd parties and users, just like eg. adblock blacklists and whitelists. For example, Python devs might maintain a list of search terms and associated urls that g…

> to guess for you based on your search history, you can opt-in to download and apply such lists of ranking modifiers to your user profile

pro-privacy does not sit well with terms such as search history and user profile

Re: We can do better than DuckDuckGo

#124

Post by meerita »

I'm an old-time user of DDG. I agree with the 3 points. I feel like I'm using a shitty car in a world where everyone runs a ferrari.

The one that strikes me most is the results. I feel like DDG doesn't search the entire internet, like there's zillions of pages there waiting to be indexed, even old websites, but the results i get are so poor.

Even with this handicap, I still use over INSERT YOUR AD HERE Google Search.

Re: We can do better than DuckDuckGo

#125

Post by Jtsummers »

Shared404 wrote: post #99
jedimastert wrote: post #50

> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…

> Who chooses the white list, and why should I trust them? Is it democratically chosen? You could have user compiled lists of sites to show in search results. Let the users pick the lists they want to see, and communities can create and distribute lists within themselves.

That’s what directory sites offered once upon a time. It was a pretty good way to discover new content back then. I spent a lot of time on dmoz when I wanted to find information about various topics.

Re: We can do better than DuckDuckGo

#126

Post by RileyJames »

Shared404 wrote: post #99
jedimastert wrote: post #50

> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…

> Who chooses the white list, and why should I trust them? Is it democratically chosen? You could have user compiled lists of sites to show in search results. Let the users pick the lists they want to see, and communities can create and distribute lists within themselves.

Great idea, but why build a search engine at all in this case? You can use DDG + your filter and see only the results from your whitelist.

Could easily be implemented for any current search engine.

To a large extent, this is what you already do when you view a page of search results. Filter them based on your understand of what sites / results hold value.

Re: We can do better than DuckDuckGo

#127

Post by djsumdog »

jron wrote: post #60

SEO is crushing the utility of Google. It is pretty telling when you need to add things like site:reddit.com to get anything of value. Harnessing real user experiences (blogs, etc) is the key to a better search engine. This model unfortunately crumbles under walled gardens which is increasingly the preferred location of user activity.

> Harnessing real user experiences (blogs, etc)

This is what we need more than anything. More independent blogs. The ability to search events now, or 10 years ago, mass indexing of RSS feeds, etc.

A general search engine is kinda way out of the ballpark for now. But you could specialize for long form blogs, from all sides, hard-left, hard-right, women in tech, white supremacists, all the extremes and moderates.

I've love to have an interface to search a topic and see what all kinds of people have posted long form, without commentary or Twitter/Facebook bullshit "Fact checking" notices. I what to see what real writers are seeing across the spectrum on a given topic for the week or month.

Re: We can do better than DuckDuckGo

#129

Post by dumbfounder »

AsyncAwait wrote: post #115
dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

I think that is why the idea would be to have the tier 1 sites so you don't have to index as much.

I said ISN'T a pile of garbage :)

Re: We can do better than DuckDuckGo

#130

Post by ablanco »

Analemma_ wrote: post #100

I spent seven years working at Bing, and I can tell you that this guy is massively, hugely underestimating the difficulty of this problem. His repeated "it's easy! You just have to..." suggestions are absurd. This is typical HN content where someone with no domain expertise swaggers in and assumes everyone in the space must be idiots, and that only he can save the day. Trust me, there is not a ton of potential "just…

Given your experience, What's your opinion about ddg result quality?
Post reply on HN