Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

191–200 of 396 posts

Re: The world needs a non-profit search engine

#191

Post by flexie »

jka wrote: post #154
dewey wrote: post #137

Earlier quoted context omitted.

Isn't that one more data sets for ML other research purposes instead of a highly up to date search index (For example with news from a few minutes ago).

Roughly speaking, yep - Common Crawl provides a sizable chunk of web data (420 TiB uncompressed, over 3 billion unique URLs, as of May 2022; historic statistics here[1]), and is updated on monthly basis. Not near-real-time, true, albeit relatively fresh. A question to ask could be: how often do users care about information from a few minutes ago, compared to information that has been available for a longer duration o…

Isn't that more a question of adding to the mix frequent scraping of

- a few thousand news-sites (like nyt.com, bbc.co.uk),

- a few thousand very popular blogs (based on what influencers people search for),

- a handful of social media sites (e.g. Twitter),

- a few hundred databases in areas like weather, airlines, sports (like ATP for people who look for Wimbledon results today)?

Re: The world needs a non-profit search engine

#193

Post by allendoerfer »

nonameking2026 wrote: post #189
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I found the thought of adding a "yellowlist" funny, as some people are starting to find the use of "whitelist" and "blacklist" racist.

Re: The world needs a non-profit search engine

#194

Post by lotsofpulp »

amelius wrote: post #182
sevazhidkov wrote: post #174

Earlier quoted context omitted.

Given that enough people use this heuristic, there will be companies focusing on earning karma on HN, writing comments and voting for products they are getting paid for.

Trust can work both ways. They'd have to be really careful to not lose the earned trust. On the way, they would have to write a lot of high quality HN posts. I'd say this model is a win over current spamming and fake reviewing practices.

The other side of them having to write a lot of high quality HN posts is me having to read and evaluate more HN posts trying to game me. That is work, and if I sense a lot of it (like I did on Reddit), I will leave, and I suspect others would too.

Playing defense is exhausting when playing offense is extremely cheap.

Re: The world needs a non-profit search engine

#195

Post by daoudc »

soperj wrote: post #24

I really like your idea to have users help you crawl the web. I just don't like what your extension is asking for when I try to install it: - Access your data for all websites - Monitor extension usage and manage themes

Feel free to read the source code if you're concerned about what it does!

Re: The world needs a non-profit search engine

#196

Post by closedloop129 »

Sujeto wrote: post #178

Most of the time you want answers from a certain place, be it reddit, or stackoverflow. It's usually easy enough to add the site, by simply writing its name, but it could be easier. I'm thinking right now this: - Compile a browser without the cross origin limitations - Make a site that uses iframes with all the answer-providing websites in it - Simply focus the text input in the site/iframe you want and search away -…

Why do you need to show the sides in iframes? You could submit your query to your target sites like this: https://searchaggregate.com/ [1]

[1] https://news.ycombinator.com/item?id=31722417

Re: The world needs a non-profit search engine

#197

Post by rambambram »

labrador wrote: post #77

I don't know what the world needs but I need a personalized search engine. I would like to filter out anything to do with sports. I would like filter articles that contain marketing jargon and technobabble. I would like to filter articles written below high school grade level. And so on.

> I would like to filter out anything to do with sports.

Amen. How many times I searched for an astronaut's name - or any other person with significance - and instead I got results for some pro sports guy.

I guess 'significance' is subjective and I should search more specific.

Re: The world needs a non-profit search engine

#198

Post by DoctorDabadedoo »

blowski wrote: post #190
kreeben wrote: post #180

Earlier quoted context omitted.

Wouldn't the world become a little bit better if they did? Earning karma on HN is not the easiest thing. I would hate to speak for all of us but collectively, don't you think we have a pretty good marketspeak alert system here?

I’ve got 10k karma by making pithy criticisms of Node.js and Kubernetes. Do you therefore trust my opinions on US healthcare?

Show us the way, reasonable man.

Re: The world needs a non-profit search engine

#199

Post by daoudc »

happymellon wrote: post #56

First thing I try is: Star trek imdb First result is startrek.com, second result is Star Trek into Darkness IMDb but 3rd is xkcd. It then goes off into Q and William Shatner Wikipedia links and Muppet Movie IMDB in Russian. I tried putting a plus in front of IMDB and quoting Star Trek. It doesn't seem to be able to find Star Trek on IMDB. I admire the concept, and it is extremely fast.

Our index is still very small. Help make it bigger!

Re: The world needs a non-profit search engine

#200

Post by slazaro »

nonameking2026 wrote: post #189
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

99%+ of people wouldn't use this, they would just try the search engine, see that the results suck, then stop using it.

It could be useful for power users, though.

Post reply on HN