Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

251–260 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#251

Post by guerrilla »

AdamN wrote: post #241

IMHO what you should try is excluding all sites with excessive third-party cookies, sluggish performance, and too many ads. That will slice the index down by 80% probably but it would be a really nice thing to see. It might push out low quality SEO results for a couple of years.

This is the solution. Google and DuckDuckGo should be doing this too (and make exceptions if they need to so that they don't collapse). We have to incentivize the good behavior and create an environment where people actually compete on the properties we want and not horseshit.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#252

Post by MicahKV »

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedicated to thwarting your defensive efforts. Even Google struggles to keep up with this.

However, they are also systematically feeding you their footprint lists. I imagine you could put together a footprint blacklist pretty quickly, and just stop returning results for any obvious spam queries like those containing "powered by wordpress".

It's not a very elegant solution I'll admit. It won't stop the bots from trying, and you may have to circle back periodically to add new footprints as they surface. But it's a potentially quick and easy way to stop rewarding their efforts, and the blackhat world is pretty used to burning out their resources so hopefully they will figure out it's a dead end and move on.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#253

Post by omoikane »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I think the Dead Internet Theory bit is just a bait to get more comments. It's a bit of a stretch to conclude that the internet is mostly robots just because one website sees mostly robots. This extrapolation would be convincing if that one website is a high ranking website that sees a lot of traffic, but searchmysite.net does not appear to be one of the top websites.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#254

Post by m-i-l »

wibyweb wrote: post #144

In late April up to now, Wiby (a small mostly unheard of search engine) began having the exact same issue. Tens of thousands of the exact same type of "powered by..." requests coming from thousands of IPs. They are using a tool called QHub.

Thanks for wiby.me. I have seen QHub coming up in the scraping footprints, but my assumption has been that the footprint query is looking for Question and Answers sites powered by QHub containing their targeted terms, e.g. because there's a known vulnerability with QHub that their scripts can exploit to auto-post backlinks or whatever it is they do. There are lots of other hosting tools, other than QHub, that come up in the footprints as well. I found some lists of footprints by doing an internet search for one of them: "Designed by Mitre Design and SWOOP".

Re: Almost all searches on my independent search engine are now from SEO spam bots

#255

Post by ufmace »

maxwelldone wrote: post #104
mywaifuismeta wrote: post #29

Earlier quoted context omitted.

I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.

Back in 2000s Google used to be the place for any type of search (IIRC). Now, I've been conditioned to use it only for specific use cases, mostly for convenience. Some examples include: 1. Anything programming related (searching for man pages, error codes etc) is straightforward. (I do have some UBO filters to exclude SO copycats) 2. Utility stuff like currency conversion, finding time in another city, weather etc. W…

Programming-related stuff seems to have gotten a lot worse in the last couple of years. Now most terms, at least for common things, return a ton of blogspam, when the official docs or SO are usually the best source.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#256

Post by marginalia_nu »

dylan604 wrote: post #227
marginalia_nu wrote: post #209

Earlier quoted context omitted.

Why would I need to generate money to stay relevant?

The first relevant was the wrong word. sustainable would be more appropriate. on the assumption that hosting the search engine isn't free, and unless it is supported by a generous benefactor it will need to have a way of generating money to keep the servers running.

I'm self hosting so my operational cost is like $50/mo.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#257

Post by m-i-l »

marginalia_nu wrote: post #173
marginalia_nu wrote: post #51

I had to put my search engine behind Cloudflare to deal with this. Like the volume grew to about 10x the traffic I saw sitting at the front page of Hacker News for a full week.

This is the rate of rejected HTTP requests I'm seeing at this point: https://www.marginalia.nu/junk/spam.png Real search traffic is about half that.

Thanks V. I'm seeing a similar number of problem search requests (although nowhere near as many real search requests:-), so it is probably the same "SEO practitioners" running the same "scraping footprints" against different search engines around the same time.

I was kind-of hoping that somewhere in this discussion there would be an "And the answer to your problem is...", but I suppose it is a very specific problem which only a search engine would encounter. I think the Cloudflare solution you have is probably the best to block the requests as early as possible. The reverse proxy config[0] I've got seems to be mostly holding out for now though.

[0] https://github.com/searchmysite/searchmysite.net/issues/55

Re: Almost all searches on my independent search engine are now from SEO spam bots

#258

Post by kodah »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

Agreed, the general internet is not dead, but the majority of internet users are on Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc.

From my perspective, we onboarded a lot (if not most) people to the internet after 2007 (the explosion of social media). People sticking to big sites really speaks to an inability to explore the larger internet and a lack of knowing why you would even want to.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#259

Post by [deleted] »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

[deleted]

Re: Almost all searches on my independent search engine are now from SEO spam bots

#260

Post by john-radio »

Since everyone in this thread wants to jump down OP's throat about the quality of his web site, another interesting search engine is millionshort.com, which allows you to filter out the top N web sites from the results of your search. It's a great tool for looking past sites with good SEO; all you have to do is fiddle with the value of N.

For example, searching for "electronic music box" as /u/ajnin suggested, with the top 100K web sites removed from the results, filters out the following:

> These 23 sites were removed from your results:

> alibaba.com (1 result removed)

> aliexpress.com (1 result removed)

> allaboutcircuits.com (1 result removed)

> amazon.com (2 result removed)

> apple.com (1 result removed)

> bestreviews.com (1 result removed)

> ebay.com (1 result removed)

> etsy.com (2 result removed)

> facebook.com (1 result removed)

> instructables.com (2 result removed)

> lightinthebox.com (2 result removed)

> lumberjocks.com (1 result removed)

> mapquest.com (1 result removed)

> reverb.com (1 result removed)

> twitter.com (1 result removed)

> wikipedia.org (1 result removed)

> yelp.com (1 result removed)

> youtube.com (2 result removed)

And the top result ends up being https://midiguy.com/.

Post reply on HN