Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Messing with scraper bots

herman.bearblog.dev

Post reply on HN
Print view View original

71–80 of 91 posts

Re: Messing with scraper bots

#71

Post by dotancohen »

simondotau wrote: post #67
akoboldfrying wrote: post #63

Earlier quoted context omitted.

That was my first thought too -- but then why would the bot company care about a few false positives? I suppose it could have an impact if 30% of all, say, Coca Cola mentions on the web came from that site, but then it would have to be a very big site. I don't think the bot company would notice , let alone care, if it was 0.01% of the mentions.

Everyone’s definition of “big” is different, but back then it was big enough to get its own little island in a far corner of XKCD 802. https://xkcd.com/802/

Diaspora?

Re: Messing with scraper bots

#72

Post by akoboldfrying »

akoboldfrying wrote: post #60
georgefrowny wrote: post #13

Earlier quoted context omitted.

There's a difference between putting information easily online for your customers or even people in general (eg as a hobby), and working in concert with scraping for greater visibility via search, and giving that work away, or at a cost, to companies who at best don't care and possibly may be competition, see themselves as replacing you or otherwise adversarial. The line is "I technically and able to do this" and "I…

> The line is "I technically and able to do this" and "I am engaging with a system in good faith". This is where the line should be, always. But in practice this criterion is applied very selectively here on HN and elsewhere. After all: What is ad blocking, other than direct subversion of the site owner's clear intention to make money from the viewer's attention? Applying your criterion here gives a very simple concl…

I see downvotes, but no counterarguments.

Does anyone have a counterargument?

Re: Messing with scraper bots

#73

Post by simondotau »

grishka wrote: post #62
simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

Thank you very much for the observation about headers. I just looked closer at the bot traffic I'm currently receiving on my small fediverse server and noticed that it's user agents of old Chrome versions but also that the Accept-Language header is never set, which is indeed something that no real Chromium browser would do. So I added a rule to my nginx config to return a 403 to these requests. The amount of these pe…

The important thing is to be aware of your adversary. If it’s a big network which doesn’t care about you specifically, block away. But if it’s a motivated group interested in your site specifically, then you have to be very careful. The extreme example of the latter is yt-dlp, which continues to work despite YouTube’s best efforts.

For those adversaries, you need to work out a careful balance between deterrence, solving problems (e.g. resource abuse), and your desire to “win”. In extreme cases your best strategy is for your filter to “work” but be broken in hard to detect ways. For example, showing all but the most valuable content. Or spiking the data with just enough rubbish to diminish its value. Or having the content indexes return delayed/stale/incomplete data.

And whatever you do, don’t use incrementing integers. Ask me how I know.

Re: Messing with scraper bots

#74

Post by quesera »

kijin wrote: post #36
palsecam wrote: post #35

Earlier quoted context omitted.

I do something quite similar with nginx: # Nothing to hack around here, I’m just a teapot: location ~* \.(?:php|aspx?|jsp|dll|sql|bak)$ { return 418; } error_page 418 /418.html; No hard block, instead reply to bots the funny HTTP 418 code ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... ). That makes filtering logs easier. Live example: https://FreeSolitaire.win/wp-login.php (NB: /wp-login.php is Wor…

nginx also has "return 444", a special code that makes it drop the connection altogether. This is quite useful if you don't even want to waste any bandwidth serving an error page. You have an image on your error page, which some crappy bots will download over and over again.

Beware of nginx 444 if your webserver is behind a load balancer.

The LB will see the unresponded requests and think your webserver is failing.

Ideal would be to respond at the webserver and let the LB drop the response.

Re: Messing with scraper bots

#75

Post by kalkin »

lavela wrote: post #8
s0meON3 wrote: post #6

What about using zip bombs? https://idiallo.com/blog/zipbomb-protection

"Gzip only provides a compression ratio of a little over 1000: If I want a file that expands to 100 GB, I’ve got to serve a 100 MB asset. Worse, when I tried it, the bots just shrugged it off, with some even coming back for more." https://maurycyz.com/misc/the_cost_of_trash/#:~:text=throw%2...

Ah cool that site's robots.txt is still broken, just like it was when it first came up on HN...

Re: Messing with scraper bots

#76

Post by mnau »

NoiseBert69 wrote: post #3

Hm.. why not using dumbed down small, self-hosted LLM networks to feet the big scrapers with bullshit? I'd sacrifice two CPU cores for this just to make their life awful.

He addresses that. Basically, there are gatekeepers and if you get on the wrong side of them, only manual intervention can save you. And we all know how Google loves providing a human to resolve problems.

> I came to the conclusion that running this can be risky for your website. The main risk is that despite correctly using robots.txt, nofollow, and noindex rules, there's still a chance that Googlebot or other search engines scrapers will scrape the wrong endpoint and determine you're spamming.

Re: Messing with scraper bots

#77

Post by grishka »

simondotau wrote: post #73
grishka wrote: post #62

Earlier quoted context omitted.

Thank you very much for the observation about headers. I just looked closer at the bot traffic I'm currently receiving on my small fediverse server and noticed that it's user agents of old Chrome versions but also that the Accept-Language header is never set, which is indeed something that no real Chromium browser would do. So I added a rule to my nginx config to return a 403 to these requests. The amount of these pe…

The important thing is to be aware of your adversary. If it’s a big network which doesn’t care about you specifically, block away. But if it’s a motivated group interested in your site specifically, then you have to be very careful. The extreme example of the latter is yt-dlp, which continues to work despite YouTube’s best efforts. For those adversaries, you need to work out a careful balance between deterrence, solv…

In my particular case, I don't mind the crawling. It's a fediverse server. There is nothing secret there. All content is available via ActivityPub anyway for anyone to grab. However, these bots specifically violated both robots.txt and rel="nofollow" while hitting endpoints like "log in to like this post" pages tens of times per second. They were just wasting my server's resources for nothing.

Re: Messing with scraper bots

#78

Post by simondotau »

DamnInteresting wrote: post #70
simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

I did something similar with someone who was using my site’s donation form to test huge batches of credit cards numbers. I would see hundreds of attempted (and mostly declined) $1 donations start pouring in, and I’d block the IP. A little while later it would restart from another IP. When it became clear they were not giving up easily, I changed tack: instead of blocking them, I would return random success/failure me…

Was it always $1? If I was the attacker, surely you’d pick a random number. My guess is that $1 donations would be an outlier in the distribution and therefore easy to spot.

It’s also interesting that merchants (presumably) don’t have a mechanism to flag transactions as being >0% chance of being suspect. Or that you waive any dispute rights.

As a merchant, it would be nice if you could demand the bank verify certain transactions with their customer. If I was a customer, I would want to know that someone tried to use my card numbers to donate to some death metal training school in the Netherlands.

Re: Messing with scraper bots

#79

Post by testing22321 »

bot403 wrote: post #39
warkdarrior wrote: post #34

Earlier quoted context omitted.

Make sure to block your own IP address to minimize the chance of a social engineering attack.

Include 127.0.0.1 as well just in case they get into the server.

The server I have not built yet is my most secure one yet!

Re: Messing with scraper bots

#80

Post by amypetrik8 »

VladVladikoff wrote: post #32

This is a fundamental misunderstanding of what those bots are requesting. They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities. They probably immediately stop reading after receiving a http response code and discard the remainder of the request packets.

> They aren’t parsing those PHP files, they are using their existence for fingerprinting — they are trying to determine the existence of known vulnerabilities.

So would the natural strategy then be to flag some vulnerability of interest? Either one typically requiring more manual effort (waste their time), or one that is easily automated so as to trap a bot in a honeybot i.e. "you got in, what do next? oh upload all your kit and show how you work? sure" see: the cuckoos egg

Post reply on HN