Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

261–270 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#261
post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I searched "best dress shoes reddit" as a test, and just got a random list of websites that had the word "shoes" on the page somewhere, including a Dinosaur Comic from 2008.

So... yeah. Won't exactly be my first choice of search engine in the future.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#262

Earlier quoted context omitted.

Agreed. > If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam. The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that. Users of marginalia (aut…

> The "general" Internet is not dead. For some things it is. Good luck getting a non-sponsored/SEO-gamed review of a kitchen appliance or particular vacation mode such as a cruise. It's flabbergasting. Most times I just stick "inurl:reddit.com" in my search and try to get discussion threads about the thing I'm researching, but even that's getting filled up with shills.

I think search engines are broken, but the Internet itself is probably not "dead". It's just our accessibility to that information. That's not super helpful until we have better search engines (which steer us away from this SEO stuff), but the good news is that building a better search engine is easier than resurrecting the Internet. In particular, there's a good chance that a niche, naive search engine might be able to significantly improve accessibility (e.g., high rankings for pages that answer user queries in the fewest bytes).

Re: Almost all searches on my independent search engine are now from SEO spam bots

#263
post #86
post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

This isn't indexing by search engine spiders, which are usually fairly benign and easy to identify with user agent etc. This is searches for "scraping footprints" executed en mass by "SEO proxy farms", which are designed to be very difficult to detect (e.g. originating from globally distributed residential IPs, quite possibly ordinary home user's machines which have been compromised). The main giveaway that something…

That's funny to me, because if I'm searching for something that would have been around between 2004-2012, I'll often append "Powered by phpBB" (or other software) to find posts about it on forums.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#264
People will only use your product if they know about it and perceive value in it. How do people know about it, and why would they want to use it?

On "Most of the tiny number of real users have come from links posted to places like Hacker News, and there is almost no organic traffic from other search engines" - Organic traffic comes from word of mouth. Are people talking about your site? If they're not, you're not gonna see organic traffic. You could do what others do and pay some influencers to advertise your site, but that's expensive and not as scalable as "real" buzz. Is your product exciting or controversial? If not, why would people talk about it?

Your homepage's tag line is "Open source search engine and search as a service for personal and independent websites." A regular person's eyes would glaze as they try to figure out what this means. Given some time they might put together the words "search engine" and "personal" and "websites" and figure this is a blog search engine. So just say that.

The "Newest Pages" section is a fun novelty, but after a few minutes the novelty wears off.

The "Browse Sites" section is almost useful. Next to the list of sites I see some tags. Why isn't a heatmap of the tags the first thing I see? That would be way more useful than a paginated list of random sites.

Your "About" page lists "community-based approach to content curation". This is the most exciting aspect of the whole endeavor, so add that to your front page blurb ("Community search engine"). You would probably do well to build a real community around it, for example with a forum or chat system (GitHub Discussions does not count). A SubReddit would be an easy way to bootstrap this and later move it to your own hosted forum.

You'll probably need a very complicated moderation system if this thing takes off.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#265
post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I searched "best dress shoes reddit" as a test, and just got a random list of websites that had the word "shoes" on the page somewhere, including a Dinosaur Comic from 2008. So... yeah. Won't exactly be my first choice of search engine in the future.

It doesn't index sites like Reddit, so, not too surprising Reddit wasn't in the result.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#266
post #188

Earlier quoted context omitted.

Well, it's worse than that. The whole schtick is that it's only pure, real content by folksy people like us. The top reason to use it on the about page is: Indexes only user-submitted sites with a moderation layer on top, for a community-based approach to content curation, rather than indexing the entire internet with all of its spam, "search engine optimisation" and "click-bait" content. So I tried searching [kotlin…

Thanks for your feedback. It is just the home page which is moderated before indexing (and reviewed annually). When https://personalsit.es/ was listed it looked legitimate, but agreed the results for that site look infected with spam now. I've found at least one other site today where the home page and blog look genuinely legitimate, but which has a complete spam subdomain, quite possibly the victim of a subdomain ta…

Your problem is that SEO are under no obligation to be truthful with you, and will likely pull bait and switches as far as making accounts if it ever seems like your site will catch on.

Note, I nearly spit my food the first time I was at lunch and someone was talking about SEO a few tables away...oh a decade or so ago now. It's sad it's gotten this bad.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#267
post #231
post #199

Earlier quoted context omitted.

Hi, "this guy" here:-) If people come to a site but don't come back then it is reasonable to conclude that "it's a bad website", but as the blog entry put it "without any real users in the first place it is hard to gauge whether people like it or not". Note also that it isn't intended to be a general purpose search engine, but a niche search engine to try and find some of the fun and interesting content, e.g. relatin…

How exactly is a "general purpose search engine" different than a "search engine to try and find some of the fun and interesting content"?

The general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?" and "What time is it?". However, niche search engines only search specific parts of the internet, and only claim to be able to deliver results relating to their specific topic, e.g. you wouldn't ask the search on a car forum what the weather is today.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#268

Earlier quoted context omitted.

I fear that Google also has a conflict of interest here. A lot of these non optimized sites are not interested in making money via ads. So Google wouldn't profit additionally from leading people there. And a lot of people (myself often times included) are looking for a quick answer. A good enough answer. So good enough, SEO optimized is being surfaced. The result of an optimization war on both sides combined with the…

Maybe back to Yahoo model of the 90s? Manually created collection of curated links?

Yes. We have enough users now.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#269

Earlier quoted context omitted.

I just search Kagi, Google, and DDG for "Astral Codex Ten" and it was the first result on each.

Ironically the Kagi search engine is not in the first few results in Google when you search Kagi (at least in Thailand) And when I did make it to the site, it looks like I have to sign up to use it? I'm not sure putting a locked gate in front of a search engine in 2022 makes sense but okay

The whole concept of kagi is to be a paid service (is still in beta and for now it's free AFAIK), so you pay money instead of having ads or the search engine selling your data, use the service that suits best to your purposes and philosophy.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#270
post #212

Earlier quoted context omitted.

https://plausible.io/privacy-focused-web-analytics You surf with JS off because of sites abusing their users' data. This is not it.

Collecting data that a user doesn't want collected is abuse. It doesn't matter what you do with it.

Oof. Hard disagree on that one, way too black & white of a position for me in the face of such a broad concept as "data".
Post reply on HN