Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

101–110 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#101

Post by lysecret »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I think this is a very "consumer focused" take. Yes. A lot of interesting people data is now "locked" behind these aggregators and platforms (and also hard to handle because of GDPR). But most interesting company data is still out there.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#102

Post by simion314 »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

This is not true, maybe for a subset of Internet users.

For example you have Wikis and forums. Wikis are good for communities that are passionate about a topic and they collaborate on buidling content for their passion. Reddit is a valid alternative to forums but if the community s older and has members that are technical competent then they usually have the forum customized for their purpose and the forum will continue to exist , especially if you want to avoid some third party censorship.

I never ever search for something and found answers on Facebook, sometimes very rare I find something that points to Instagram blogs/posts but never Facebook.

Probably depends on your location and what you search for, so it might be possible that 99% of your Internet consumption is satisfied by 5-10 websites.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#103

Post by f0xJtpvHYTVQ88B »

albatrosstrophy wrote: post #6

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

Brave Search recently implemented "discussions". From what I've seen it is mostly Reddit results but StackExchange also can appear there.

https://searchengineland.com/brave-search-discussions-383706

Re: Almost all searches on my independent search engine are now from SEO spam bots

#104

Post by maxwelldone »

mywaifuismeta wrote: post #29
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.

Back in 2000s Google used to be the place for any type of search (IIRC).

Now, I've been conditioned to use it only for specific use cases, mostly for convenience. Some examples include:

1. Anything programming related (searching for man pages, error codes etc) is straightforward. (I do have some UBO filters to exclude SO copycats)

2. Utility stuff like currency conversion, finding time in another city, weather etc.

Where Google has really fallen behind is in multimedia search. Not sure if it's due to copyright issues or not but Bing and Yandex provide way better service in this regard.

Not to mentions the "reddit" suffix I need to add to any search that even remotely calls for public opinion. In many cases, Google is just a shortcut to take me to the relevant subreddit.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#105

Post by jrussbowman »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

"Want to become rich? Make a search engine which indexes the fresh relevant data from the big siloed websites, and ignores the general dead Internet."

Did that to some degree. Unscatter.com pulls from reddit and twitter to source links.

I found reddit only created an echo chamber bubble of obvious bias and twitter only diluted it a little.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#106

Post by ratww »

marginalia_nu wrote: post #84
indigochill wrote: post #72

Earlier quoted context omitted.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

I do a traditional web crawl and exclude anything that looks too much like it wants a high google ranking. Nothing to it.

This might be controversial, but I wish Google would exclude those websites too.

Google started punishing keyword spam, then it started punishing black-hat comment spam. Even Youtube backtracked on the "videos have to be 10 minutes to rank".

I wish they would do the same for carefully manicured SEO content farms too, as those sites are causing a harm worse than keyword-spammer sites did.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#107

Post by Elvie »

Gravityloss wrote: post #95
shortformblog wrote: post #92

Earlier quoted context omitted.

I think this is a tad reductive, but I will say that we sure let a lot of big companies convince a huge portion of the population to create all of their content on platforms that they have no real control over. The problem is, many of them didn’t realize this was a problem until recently. That said, plenty of exciting stuff is happening outside of the walled garden, as long as you know how to find it.

And not only did this happen already over a decade ago, a lot of the current internet users have never known anything else. We had a discussion with coworkers and somebody mentioned irc. Explaining to younger colleagues what it was and that it was not a product of a company, but operators had servers that formed a network, and it was more like infrastructure. Felt weird.

isn't Discord a bit like IRC used to be?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#108

Post by buzzwords »

I have had very interesting conversations with people who are "casual" users of internet. They are still finding the results of the likes of Google, bing and duckduckgo perfectly suitable. Maybe it's most of us here who have different needs to what's available.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#110

Post by aceazzameen »

FargaColora wrote: post #74
airstrike wrote: post #71

Earlier quoted context omitted.

> Reddit doesn't let you fully expand threads when you're unlogged. that's what old.reddit.com is for!

old.reddit will be gone soon, it is inevitable. Especially once they go public.

Yup. It's bound to happen. And when it does, Reddit will no longer exist in my eyes.
Post reply on HN