301–310 of 391 posts
So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…
Will we ever see the return of hand curated directories of websites like the old days, categorized by topics and approved by human review?
Wikipedia, maybe? The greater problem of curation is that it doesn't scale, and you need immense human effort to survey and curate both the breadth of questions -- what's a good table saw? what aspects of Egyptian culture were exported back to Greece? is HDPE plastic safe? give me some punk music. -- and also the breadth of answers, both every website and every type of table saw. The lesser is that you cannot curate…
Earlier quoted context omitted.
Agreed. I suspect that this is an arbitrage game on the part of the SEO spammers. Each search is cheaper for them than it is for a competitor who's using a major search engine with more extensive anti-spammer protections, and that difference equals $$$. A captcha doesn't have to be an unbeatable solution. It just has to provide enough of a barrier to equalize the cost.
I'm not so sure about this. The spammers goal is to build up as big a list of link spam targets as possible. If one spammer chooses to only scrape minor engines and another only major engines, the one scraping the major engines will probably come out on top despite the higher cost. Whoever is abusing OP's search engine is likely doing it to supplement the data they are already scraping from the major engines. For OP,…
Earlier quoted context omitted.
Or put those query results behind an anti-bot/"capcha" test.
That would probably help, but it's also a continuation of the cat and mouse game. There are plenty of captcha breaking services out there, it only cost about $1 to programmatically solve 1000 captchas.
1. Almost all searches on my independent search engine are now from SEO spam bots
2. In summary, if they break through the current reverse proxy level protection, options include an invisible ReCAPTCHA (but given I’ve sometimes 160,000 requests a day I’d be well over the 1,000,000 a month free tier limit), requiring JavaScript as per the web analytics or some Cross Site Request Forgery style protection (but those would place much more load on the servers), or CloudFlare (but the searchmysite.net spider is still currently blocked by CloudFlare as per Some of the challenges of building an internet search)
3. If you were into conspiracy theories you could claim that the major search engines were trying to stifle the competition, but a more realistic explanation is simply that searchmysite.net is being drowned out by SEO spam
4. If I’d had a decent amount of real users visiting and never returning I could reasonably conclude that updating the blog wasn’t the most productive use of my time and effort, but without any real users in the first place it is hard to gauge whether people like it or not
My own independent search engine, https://www.locserendipity.com, is seeing similar trends.
Earlier quoted context omitted.
This is the rate of rejected HTTP requests I'm seeing at this point: https://www.marginalia.nu/junk/spam.png Real search traffic is about half that.
Thanks V. I'm seeing a similar number of problem search requests (although nowhere near as many real search requests:-), so it is probably the same "SEO practitioners" running the same "scraping footprints" against different search engines around the same time. I was kind-of hoping that somewhere in this discussion there would be an "And the answer to your problem is...", but I suppose it is a very specific problem w…
Earlier quoted context omitted.
I'm not so sure about this. The spammers goal is to build up as big a list of link spam targets as possible. If one spammer chooses to only scrape minor engines and another only major engines, the one scraping the major engines will probably come out on top despite the higher cost. Whoever is abusing OP's search engine is likely doing it to supplement the data they are already scraping from the major engines. For OP,…
It might be a better idea to return low quality results than nothing at all. The idea is that it's pretty obvious when the bot is banned when it receives no results at all. Having to look at the results manually to determine whether one is banned is a much more time consuming endeavor.
If anything, it might be best to return a page that explicitly states "Sorry, this search engine no longer supports SEO footprint search queries."
*edit for typo & wording
Earlier quoted context omitted.
That would probably help, but it's also a continuation of the cat and mouse game. There are plenty of captcha breaking services out there, it only cost about $1 to programmatically solve 1000 captchas.
... and there are the "click farms" with human beings.
Earlier quoted context omitted.
I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.
I find one of the best ways to find interesting content on specific subjects using Google is now to start blocking all their top returns (a lot of SEO spam). This is somewhat tedious (lots of -site:seospam.com) and Google doesn't like automated queries. However, a few rounds of this often turns up interesting content down low in the search results. Just don't take what's on offer on page one of search results, basica…
You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.