We can do better than DuckDuckGo • Hacker News Board

Re: We can do better than DuckDuckGo

#161

Post by djsumdog » Tue, Nov 17, 2020, 10:24 PM UTC

eqv wrote: post #70

mekster wrote: post #52

Earlier quoted context omitted.

Author didn't even DDG to find this out?

Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998

I don't like to criticize the author. We all have good takes and bad takes and really for a single post, you should address the argument. Digging up the past is part of what's making the world worse.

That being said, I do see a valid reason for bringing up his history of bad takes. I use to respect Devault. He banned me on the Fediverse because he disagreed with me being against defunding the police and against critical race theory.

I find some of stuff interesting, and I agree with more AGPL and more real open source development. I'd even say I'm jealous that he can actually fund himself off of his FOSS projects and do what he loves.

But I do agree, he does have a lot of questionable takes. He seems to love Go and hate Rust, hate threads for some reason, and has a lot of RMS style takes. Not all of them are bad, and hardcore people can help you think.

As far as this post goes, I do think search is pretty broken. I think a better solution is more specialized search. Have a web tool just for tech searching that does StackExchange sites, github, blogs, forums, bug trackers and other things specialized to development.

Another idea would be an index that just did blogs, do you can look up any topic and see what people are writing about long form for the current month. Add features to easily see what people were saying 5 or 10 years ago too. There is a ton of specialized work there, in filtering blog spam, making sure you get topics from all sides (including "banned" blogs), etc.

You use to have to go to Lycos, Yahoo, Hotbot, Excite and you'd get different results and find lots of different helpful things. We need that back. It will take some good, specialized tools, to break people from Google search.

Re: We can do better than DuckDuckGo

#162

Post by lemax » Tue, Nov 17, 2020, 10:25 PM UTC

dumbfounder wrote: post #85

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entir…

Open Street Map is a nice analogy for what could work. Aside from the open source maintenance of the map, there's also tons of corporate help in the background. Companies that are delivering OSM as a service or relying on it for their own services have an interest in making it better. MapBox, for example, apparently pays tons of people a salary who are contributing upstream to Open Street Map. If we can get an Apple/Microsoft/Other players collab maybe a venerable alternative can actually be built.

Re: We can do better than DuckDuckGo

#163

Post by djsumdog » Tue, Nov 17, 2020, 10:25 PM UTC

Eeems wrote: post #160

eqv wrote: post #70

Earlier quoted context omitted.

Drew has a longstanding history of ill-informed rants ([1] [2]) about technology. He's also quite willing to lie about the facts[3]. [1] https://news.ycombinator.com/item?id=24121609 [2] https://news.ycombinator.com/item?id=23966778 [3] https://news.ycombinator.com/item?id=24023998

Just don't give him IRC ops and then get into a private argument with him. https://www.omnimaga.org/news/omnomirc-moved-to-new-server/m...

Wow. I'm honestly not surprised. That's ... that's pretty shitty.

Re: We can do better than DuckDuckGo

#164

Post by ecommerceguy » Tue, Nov 17, 2020, 10:26 PM UTC

jedimastert wrote: post #50

> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustw…

So basically lock out any new site, regardless of content. Great idea /s

Re: We can do better than DuckDuckGo

#165

Post by CameronNemo » Tue, Nov 17, 2020, 10:27 PM UTC

ant6n wrote: post #152

djsumdog wrote: post #127

Earlier quoted context omitted.

> Harnessing real user experiences (blogs, etc) This is what we need more than anything. More independent blogs. The ability to search events now, or 10 years ago, mass indexing of RSS feeds, etc. A general search engine is kinda way out of the ballpark for now. But you could specialize for long form blogs, from all sides, hard-left, hard-right, women in tech, white supremacists, all the extremes and moderates. I've…

Its hard to get readership writing blogs these days. Thats pretty demotivating.

Also difficult to distinguish a blog from a content farm if you are just crawling the web. Any content pattern you select for would likely be quickly adopted by SEOs.

Re: We can do better than DuckDuckGo

#166

Post by moonchild » Tue, Nov 17, 2020, 10:28 PM UTC

Findx[1] was an attempt to make an opensource search engine. Today it's just another bing wrapper; but their code[2] is still available, waiting to be used as a starting point for another project.

1. https://www.findx.com/

2. https://github.com/privacore/open-source-search-engine

Re: We can do better than DuckDuckGo

#167

Post by beckingz » Tue, Nov 17, 2020, 10:28 PM UTC

eyelidlessness wrote: post #155

gregmac wrote: post #93

Earlier quoted context omitted.

I kind of hope they don't, or there is more to it than just that -- for example, a user coming back and clicking on something else counts as a downvote for the first item. Any system that ranks things purely based on votes or view counts can have a feedback loop that can amplify "bad" results that happen to get near the top for whatever reason. For web search, this would encourage results that look right from the res…

> Any system that ranks things purely based on votes or view counts can have a feedback loop that can amplify "bad" results that happen to get near the top for whatever reason. I think this is a place where Google has pretty obvious algorithm problems. For example, I’m building a personal website for the first time in many years, and obviously that means I’m doing a fair bit of looking up new or forgotten webdev stuf…

Is w3schools that bad?

for some languages, W3schools is as good a reference or better than the official documentation.

And they're definitely better than most seospam.

Re: We can do better than DuckDuckGo

#168

Post by ecommerceguy » Tue, Nov 17, 2020, 10:29 PM UTC

I'm surprised noone has rented Ahrefs database, whipped up an algorithm and called it a search engine. Besides google and microsoft, who has a bigger snapshot of the entire web (NSA not included)? Majestic maybe?

Re: We can do better than DuckDuckGo

#169

Post by claytoneast » Tue, Nov 17, 2020, 10:29 PM UTC

I wonder if you could start small on something like this. Build a proof of concept, a search engine for programmers that indexes only programming sites/material. See if you can technically do it, & if you can figure out governance mechanisms for the project. Sort of like Amazon starting with just selling books.

Re: We can do better than DuckDuckGo

#170

Post by ricardo81 » Tue, Nov 17, 2020, 10:29 PM UTC

RileyJames wrote: post #118

I’m pro privacy, but I dont have a problem with AdWords, outside of googles implementation. If AdWords targeting was purely based on the search term, I don’t mind. The search engine has to generate revenue somehow, and the revenue generated on “Saas crm” with a single click is likely to be larger than any users annual subscription. (10 - 100+ per click) I’m unclear on the ethical / privacy concerns of “AdWords” style…

FWIW it's no longer called Adwords, just Google Ads

Agree, keyword (and location, a lot of searches are for X near me) for the most part offers a way of delivering relevant ads.

Google are able to generate more income per search because of their critical mass of searches and advertisers, as well as having more data on searchers based on search history to maximise that revenue per search.