Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

71–80 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#71

Post by airstrike »

ratww wrote: post #59
captainmuon wrote: post #56

Earlier quoted context omitted.

But Twitter, Reddit, HN, and most other such places are just websites and can be indexed fine. Same with Wikipedia, which is very much a silo (they don't have regular links in text in the hypertext spirit, but only footnotes). Facebook and Instagram are more of a walled garden, like Quora, but there is a lot of junk there anyway. It's sad for the WWW, but I don't really think it is a fundamental problem for search en…

Twitter and Reddit still can be indexed, but they've also become increasingly hard to use without an account. Reddit doesn't let you fully expand threads when you're unlogged. Twitter limits the amount of things you can read and shows a modal. Both of them heavily limit usage on mobile devices without installing an app. Sure, an account is free but might require giving information you don't want to give. Twitter asks…

> Reddit doesn't let you fully expand threads when you're unlogged.

that's what old.reddit.com is for!

Re: Almost all searches on my independent search engine are now from SEO spam bots

#72

Post by indigochill »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#73

Post by lkxijlewlf »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc.

Interesting. When I search for something topical I search those sites using Google because al(most) (I don't use some like FB and insta) all those sites have really shitty search.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#74

Post by FargaColora »

airstrike wrote: post #71
ratww wrote: post #59

Earlier quoted context omitted.

Twitter and Reddit still can be indexed, but they've also become increasingly hard to use without an account. Reddit doesn't let you fully expand threads when you're unlogged. Twitter limits the amount of things you can read and shows a modal. Both of them heavily limit usage on mobile devices without installing an app. Sure, an account is free but might require giving information you don't want to give. Twitter asks…

> Reddit doesn't let you fully expand threads when you're unlogged. that's what old.reddit.com is for!

old.reddit will be gone soon, it is inevitable. Especially once they go public.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#75

Post by matheusmoreira »

kmlx wrote: post #65
matheusmoreira wrote: post #35

Earlier quoted context omitted.

The internet itself is probably gonna die soon anyway. Every country wants to impose its own laws on it. I think it'll eventually fragment into multiple segregated continental networks, if not national ones, all with heavy filtering at the borders. I'm happy to have experienced the free internet. Truly a jewel of humanity.

> I think it'll eventually fragment into multiple segregated continental networks i think it already has. the Great Firewall of China is the classic example, but I think the trend started in the west with the Right to be forgotten/right to erasure in Europe, and subsequent HTTP Status 451 Unavailable For Legal Reasons. GDPR just further cemented the split between Europe and the rest, and the new DMA & DSA regulation…

Exactly what I mean. There is no way to have an international network with national borders. Telecommunications providers have always been centralized and have always been in bed with the government. Only way we'll ever be free is if someone invents some kind of decentralized long range wireless mesh network.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#76

Post by Whiteshadow12 »

matheusmoreira wrote: post #35
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

The internet itself is probably gonna die soon anyway. Every country wants to impose its own laws on it. I think it'll eventually fragment into multiple segregated continental networks, if not national ones, all with heavy filtering at the borders. I'm happy to have experienced the free internet. Truly a jewel of humanity.

This made me sad, the optimist in me believes that some alternative will be built, that could take us back to those days. Honestly I do feel for most of my life I experienced an American Internet mostly (From South Africa), as long as one can still hop from one internet to another, in as simple a manner as possible it might not as bad as it could be.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#77

Post by [deleted] »

matheusmoreira wrote: post #35
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

The internet itself is probably gonna die soon anyway. Every country wants to impose its own laws on it. I think it'll eventually fragment into multiple segregated continental networks, if not national ones, all with heavy filtering at the borders. I'm happy to have experienced the free internet. Truly a jewel of humanity.

[deleted]

Re: Almost all searches on my independent search engine are now from SEO spam bots

#78

Post by Jleagle »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I'm getting lots of `No results found for query = xxx.`

Re: Almost all searches on my independent search engine are now from SEO spam bots

#79

Post by ratww »

airstrike wrote: post #71
ratww wrote: post #59

Earlier quoted context omitted.

Twitter and Reddit still can be indexed, but they've also become increasingly hard to use without an account. Reddit doesn't let you fully expand threads when you're unlogged. Twitter limits the amount of things you can read and shows a modal. Both of them heavily limit usage on mobile devices without installing an app. Sure, an account is free but might require giving information you don't want to give. Twitter asks…

> Reddit doesn't let you fully expand threads when you're unlogged. that's what old.reddit.com is for!

Exactly, I mentioned it. But not only it's bound to go away sometime, it's also not trivial to find to anyone who's not an expert Reddit user, unfortunately.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#80

Post by [deleted] »

TheRealDunkirk wrote: post #60
captainmuon wrote: post #56

Earlier quoted context omitted.

But Twitter, Reddit, HN, and most other such places are just websites and can be indexed fine. Same with Wikipedia, which is very much a silo (they don't have regular links in text in the hypertext spirit, but only footnotes). Facebook and Instagram are more of a walled garden, like Quora, but there is a lot of junk there anyway. It's sad for the WWW, but I don't really think it is a fundamental problem for search en…

And isn't great to get a link to Reddit or Twitter, and you click the link, and try to navigate to the comments for context or the answer, and you go to click the link to expand it, and then you get a demand to log in and install their app? Don't talk about walled gardens and not include Reddit or Twitter just because they let you look at one brick before demanding their tax.

[deleted]
Post reply on HN