Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can do better than DuckDuckGo

drewdevault.com

Post reply on HN
Print view View original

151–160 of 383 posts

Re: We can do better than DuckDuckGo

#151

Post by vorpalhex »

jedimastert wrote: post #50

> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…

I wonder if the correct answer is a blacklist for known spammy sites and the ability to turn the list off.

If I never see a pinterest link, or one of those sites that just republishes stackoverflow answers unedited, I'd be fine with it.

Of course, these systems always get abused and some political or news site will end up on it.

Re: We can do better than DuckDuckGo

#152

Post by ant6n »

djsumdog wrote: post #127
jron wrote: post #60

SEO is crushing the utility of Google. It is pretty telling when you need to add things like site:reddit.com to get anything of value. Harnessing real user experiences (blogs, etc) is the key to a better search engine. This model unfortunately crumbles under walled gardens which is increasingly the preferred location of user activity.

> Harnessing real user experiences (blogs, etc) This is what we need more than anything. More independent blogs. The ability to search events now, or 10 years ago, mass indexing of RSS feeds, etc. A general search engine is kinda way out of the ballpark for now. But you could specialize for long form blogs, from all sides, hard-left, hard-right, women in tech, white supremacists, all the extremes and moderates. I've…

Its hard to get readership writing blogs these days. Thats pretty demotivating.

Re: We can do better than DuckDuckGo

#153

Post by _underfl0w_ »

kilroy123 wrote: post #92
dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.

> People would actually use apple search.

Of course they would - it would be set as the default search on their iPhones with no clear-cut way to change it. You know, "security". The users don't know what's best for them, etc. as Apple seems to think.

Re: We can do better than DuckDuckGo

#154

Post by beefield »

Moru wrote: post #116
beefield wrote: post #109

How about a crowdsourced search engine like wikipedia or stackoverflow? Like: When you search for "kittens" you get the links that are most upvoted by the community. If nobody has ever submitted links for search term "kittens" , you get a link to selected generic search engines. And "kittens" end up into a list of words someone has searched but nobody has yet added a good result link for.

I hate to be so negative but that's just another sort of SEO problem. Someone will pay a large group of people to sit and click upvotes for their clients nonstop.

Of course there are going to be some highly debated search terms. But I think that applieas also to weikipedia and they have managed to pull it off so that it works reasonably well.

I mean, you could always put a big red badge on top of the results that says something in the line of "this search term seems to be troublesome. You may want to check qwant/ddg or maybe even google."

Re: We can do better than DuckDuckGo

#155

Post by eyelidlessness »

gregmac wrote: post #93
jedberg wrote: post #72

Earlier quoted context omitted.

I think Google sort of takes into account "votes", in that they look at the last thing you clicked on from that search, and consider that the "right answer", which they then feed back into their results. As such, they effectively have a list of "tier 1" domains.

I kind of hope they don't, or there is more to it than just that -- for example, a user coming back and clicking on something else counts as a downvote for the first item. Any system that ranks things purely based on votes or view counts can have a feedback loop that can amplify "bad" results that happen to get near the top for whatever reason. For web search, this would encourage results that look right from the res…

> Any system that ranks things purely based on votes or view counts can have a feedback loop that can amplify "bad" results that happen to get near the top for whatever reason.

I think this is a place where Google has pretty obvious algorithm problems. For example, I’m building a personal website for the first time in many years, and obviously that means I’m doing a fair bit of looking up new or forgotten webdev stuffs. It’s widely known that W3Schools is low quality/high clickbait/has a long history of gaming the SEO system. They’ve been penalized by Google’s algorithm rule changes but continue to get the top result (or even the top 3-5 results!), even with Google having a profile of my browsing habits, and knowing that I intentionally spend longer on these searches to pick a result from MDN or whatever. It seems pretty likely that W3Schools is just riding click rate to stay at the top. And it’s pathological.

Re: We can do better than DuckDuckGo

#156

Post by nostromo »

dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

> If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version.

Google pays Apple more than that every year just to set Google as the default search engine on iPhones.

In a way, Google is funding its future competitor.

Re: We can do better than DuckDuckGo

#157

Post by bmurphy1976 »

Aeolun wrote: post #131
dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

If you want a good engine there is no need to index 100B pages, since 99% of the pages are blogspam.

How are you going to identify what's blogspam and what's legitimate without indexing it all in the first place?

Re: We can do better than DuckDuckGo

#158

Post by baggachipz »

> The main problem is: who’s going to pay for it? Advertisements or paid results are not going to fly — conflict of interest. Private, paid access to search APIs or index internals is one opportunity, but it’s kind of shit and I think that preferring open data access and open APIs would be exceptionally valuable for the community.

There's no reason you couldn't allow the first N number of api hits to be free, then charge for higher tiers of access.

Re: We can do better than DuckDuckGo

#159

Post by Spooky23 »

bscphil wrote: post #47
jbay808 wrote: post #34

Earlier quoted context omitted.

Maybe instead of hard-coding these preferences in the search engine, or having it try to guess for you based on your search history, you can opt-in to download and apply such lists of ranking modifiers to your user profile. Those lists would be maintained by 3rd parties and users, just like eg. adblock blacklists and whitelists. For example, Python devs might maintain a list of search terms and associated urls that g…

I like this idea! I think the biggest difficulty with it - which is also probably the most important reason that engines like Google and DDG are currently struggling to return good results - is that the search space is just so enormously large now. The advantage of the suggestion in the blog post is that you trim down the possible results to a handful of "known good" sources. As I understand it, you'd want to continu…

I disagree; the search space is shrinking as more and more stuff moves to walled gardens like Facebook and Twitter.

Re: We can do better than DuckDuckGo

#160

Post by Eeems »

eqv wrote: post #70
mekster wrote: post #52

Earlier quoted context omitted.

Author didn't even DDG to find this out?

Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998

Just don't give him IRC ops and then get into a private argument with him. https://www.omnimaga.org/news/omnomirc-moved-to-new-server/m...
Post reply on HN