[deleted]
Messing with scraper bots
81–90 of 91 posts
Re: Messing with scraper bots
#82Earlier quoted context omitted.
The important thing is to be aware of your adversary. If it’s a big network which doesn’t care about you specifically, block away. But if it’s a motivated group interested in your site specifically, then you have to be very careful. The extreme example of the latter is yt-dlp, which continues to work despite YouTube’s best efforts. For those adversaries, you need to work out a careful balance between deterrence, solv…
In my particular case, I don't mind the crawling. It's a fediverse server. There is nothing secret there. All content is available via ActivityPub anyway for anyone to grab. However, these bots specifically violated both robots.txt and rel="nofollow" while hitting endpoints like "log in to like this post" pages tens of times per second. They were just wasting my server's resources for nothing.
My base advice is to make sure you have a very efficient code path for login pages. 10 pages per second is nothing if you don’t have to perform any database queries (because you don’t have any authentication token to validate).
Beyond that, look for how the bots are finding new URLs to probe, and don’t give them access to those lists/indexes. In particular, don’t forget about site maps. I use cloudflare rules to restrict my site map to known bots only.
Re: Messing with scraper bots
#83Earlier quoted context omitted.
> The line is "I technically and able to do this" and "I am engaging with a system in good faith". This is where the line should be, always. But in practice this criterion is applied very selectively here on HN and elsewhere. After all: What is ad blocking, other than direct subversion of the site owner's clear intention to make money from the viewer's attention? Applying your criterion here gives a very simple concl…
I see downvotes, but no counterarguments. Does anyone have a counterargument?
I think the counterargument is that a while ago ads became super annoying. They move, they grow in size, they feature nsfw things, they have weird js that annoys you when you try to leave. Perhaps some of this has toned down in recent years, but the damage is done. The ads are not good actors. It’s not as black and white as subverting or not subverting the will of the site owner.
Re: Messing with scraper bots
#84Earlier quoted context omitted.
In my particular case, I don't mind the crawling. It's a fediverse server. There is nothing secret there. All content is available via ActivityPub anyway for anyone to grab. However, these bots specifically violated both robots.txt and rel="nofollow" while hitting endpoints like "log in to like this post" pages tens of times per second. They were just wasting my server's resources for nothing.
My base advice is to make sure you have a very efficient code path for login pages. 10 pages per second is nothing if you don’t have to perform any database queries (because you don’t have any authentication token to validate). Beyond that, look for how the bots are finding new URLs to probe, and don’t give them access to those lists/indexes. In particular, don’t forget about site maps. I use cloudflare rules to rest…
Of course. My server wasn't struggling with that. I haven't benchmarked that server, but on an M1 Max, the app can easily serve hundreds of requests per second for profile pages, which is the heaviest thing an unauthenticated user can access (I cache a lot in memory, but posts, photos, and friend lists aren't among that). It was just a mild annoyance.
They discovered those URLs simply by parsing pages that contain like buttons. Those do have rel="nofollow" on them, and the URL pattern is disallowed in robots.txt, but I'd be surprised it that'd stop someone who uses thousands of IPs to proxy their requests. I don't have a site map.
Re: Messing with scraper bots
#85The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…
I did something similar with someone who was using my site’s donation form to test huge batches of credit cards numbers. I would see hundreds of attempted (and mostly declined) $1 donations start pouring in, and I’d block the IP. A little while later it would restart from another IP. When it became clear they were not giving up easily, I changed tack: instead of blocking them, I would return random success/failure me…
Yup. The only real way to stop bots is be convincing the operator that your data is poisoned.
That means you need to poison the data when you detect a bot.
Re: Messing with scraper bots
#86What about using zip bombs? https://idiallo.com/blog/zipbomb-protection
"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...
Modern browsers support brotli or zstd, which is a lot better in terms of compression. Perhaps not as good for on-the-fly compression, but static assets can get a nice compression benefit out of it.
With toxic AI scrapers like Perplexity moving more and more to headless web browsers to bypass bot blocks, I think a brotli bomb (100GB of \0 can be compressed to about 78KiB with Brotli) would be quite effective.
Re: Messing with scraper bots
#87I wonder if the abuse bots could be somehow made to mine some crypto to give back to the bills they cause
You could try to get them to run JavaScript, but I'm sure many is them have countermeasures.
Bots have had to up their compute budget to solve Anubis challenges, so perhaps it's possible if you trick the bot into thinking you're using Anubis to filter bots out.
Re: Messing with scraper bots
#88Earlier quoted context omitted.
It spams the bot with false-positives. Encourages the bot admins to denylist the site to protect the bot's signal:noise ratio.
That was my first thought too -- but then why would the bot company care about a few false positives? I suppose it could have an impact if 30% of all, say, Coca Cola mentions on the web came from that site, but then it would have to be a very big site. I don't think the bot company would notice , let alone care, if it was 0.01% of the mentions.
They dont want to feed their model with garbage data, or this data is read and revieved by real humans
I remember years-ago (2008?) I worked in a company where every mention of it was manually reviewed by someone from PR department. I imagine now the tools are even better.
Different thing is that discussion is often very low quality (forums died for multiple reasons, reddit is dying too - astro-turf gallore now)
Re: Messing with scraper bots
#89Earlier quoted context omitted.
> The line is "I technically and able to do this" and "I am engaging with a system in good faith". This is where the line should be, always. But in practice this criterion is applied very selectively here on HN and elsewhere. After all: What is ad blocking, other than direct subversion of the site owner's clear intention to make money from the viewer's attention? Applying your criterion here gives a very simple concl…
I see downvotes, but no counterarguments. Does anyone have a counterargument?
You can also argue that the advertisers have abused their position with opaque and illegal uses of personal data, security hazards, and general scummishness that they are also guilty of doing where they can technically get away with rather than what they're "supposed" to do.
Not the that two wrongs make a right, and it's definitely a bit of an argument of convenience for people who find adverts annoying. But I think most people are less opposed to the idea of advertising as popularly imagined (i.e. paper newspaper-style where you just see an advert) to support their favourite blog than they are to the current web advertising model (just by viewing the advert to get an unspecified amount of information instantly stolen and sent off to a bunch of shady companies who process it and sell it on, and don't get any way to veto it before loading a website and having the damage done).
To stretch the park analogy it might be that the park sells a licence to a company to make some cash from advertising to its visitors, which it kind of expects to be things like adverts on the benches and so on. That company then starts photographing people from the bushes, recording conversations and putting Airtags in visitors' pockets to boost the profits it makes itself. Visitors then start wearing masks, stop talking and wear clothes with zipped pockets. You can say the visitors are wrong to violate the implicit park usage agreement that they submit to the surveillance to fund the park (and advertising company), or you can say that the company is wrong to expand the original license to advertise into an invasion of privacy without even telling the visitors what they were going to do before they entered, or, indeed, during or after.
Re: Messing with scraper bots
#90This is a fundamental misunderstanding of what those bots are requesting. They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. They probably immediately stop reading after receiving a http response code and discard the remainder of the request packets.
> They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. So would the natural strategy then be to flag some vulnerability of interest? Either one typically requiring more manual effort (waste their time), or one that is easily automated so as to trap a bot in a honeybot i.e. "you got in, what do next? oh upload all…
Yeah I have been considering doing this on my honeypot.