Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Messing with scraper bots

herman.bearblog.dev

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Messing with scraper bots

#51

Post by holysoles »

ajsnigrutin wrote: post #49
holysoles wrote: post #48

Earlier quoted context omitted.

You're right, something like fail2ban or crowdsec would probably be more effective here. Crowdsec has made it apparent to me how much vulnerability probing is done, its a bit shocking for a low-traffic host.

And you'd ban the ip, their one day lease on the VM+IP would expire, someone else will get the same IP on a new VM and be blocked from everywhere. Would be usable to ban the ip for a few hours to have the bot cool down for a bit and move onto a next domain.

I was referring to the rules/patterns provided by crowdsec rather than the distribution of known "bad" IPs through their Central API.

The default ban for traffic detected by your crowdsec instance is 4 hours, so that concern isn't very relevant in that case.

The decisions from the Central API from other users can be quite a bit longer (I see some at ~6 days), but you also don't have to use those if you're worried about that scenario.

Re: Messing with scraper bots

#52

Post by firefoxd »

I had to revisit my strategy after posting about my zipbombs on HN [0]. My server traffic went from tens of thousands to ~100k daily, hosted on a $6 vps. It was not sustainable.

Now I target only the most aggressive bots with zipbombs and the rest get a 403. My new spam strategy seems to work, but I don't know if I should post it on HN again...

[0]: https://news.ycombinator.com/item?id=43826798

Re: Messing with scraper bots

#53

Post by tesin »

simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

The vast majority of bots are still failing the header test - we organically arrived at the except same filtering in 2025. The bots followed the exact same progression too. One ip, lie about the user agent, one ASN, multiple ASNs, then lie about everything and use residential IPs, but still botch the headers

Re: Messing with scraper bots

#54

Post by wvbdmp »

simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

Why do the company names chase away bots? Is it just that you’re destroying their signal because they’re looking for mentions of those brands?

Re: Messing with scraper bots

#55

Post by ronsor »

These aren't scraper bots; they're vulnerability scanners. They don't expect PHP source code and probably don't even read the response body at all.

I don't know why people would assume these are AI/LLM scrapers seeking PHP source code on random servers(!) short of it being related to this brainless "AI is stealing all the data" nonsense that has infected the minds of many people here.

Re: Messing with scraper bots

#56

Post by grishka »

Kiro wrote: post #28

I remember when you used to get scolded on HN for preventing scrapers or bots. "How I access your site is irrelevant".

It's different. I'm fine with someone scraping my website as a good citizen, by identifying themselves in their user-agent string and preferably respecting robots.txt. I'm not, however, fine with tens of requests per second to every possible URL from random IPs I'm receiving right now, all pretending to be different old versions of Chrome.

Re: Messing with scraper bots

#57

Post by akoboldfrying »

wvbdmp wrote: post #54
simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

Why do the company names chase away bots? Is it just that you’re destroying their signal because they’re looking for mentions of those brands?

I also didn't follow that part. Their step 2 seem to be a general-purpose bot detection strategy that works independently of their step 1 ("randomly mention companies").

Re: Messing with scraper bots

#58

Post by arbol »

localhostinger wrote: post #5

Interesting! It's nice to see people are experimenting with these, and I wonder if this kind of junk data generators will become its own product. Or maybe at least a feature/integration in existing software. I could see it going there.

They could be used by AI companies to sabotage each others models

Re: Messing with scraper bots

#59

Post by [deleted] »

wvbdmp wrote: post #54
simondotau wrote: post #27

The more things change, the more they stay the same. About 10-15 years ago, the scourge I was fighting was social media monitoring services, companies paid by big brands to watch sentiment across forums and other online communities. I was running a very popular and completely free (and ad-free) discussion forum in my spare time, and their scraping was irritating for two reasons. First, they were monetising my communi…

Why do the company names chase away bots? Is it just that you’re destroying their signal because they’re looking for mentions of those brands?

[deleted]

Re: Messing with scraper bots

#60

Post by akoboldfrying »

georgefrowny wrote: post #13
re-lre-l wrote: post #10

Don’t get me wrong, but what’s the problem with scrapers? People invest in SEO to become more visible, yet at the same time they fight against “scraper bots.” I’ve always thought the whole point of publicly available information is to be visible. If you want to make money, just put it behind a paywall. Isn’t that the idea?

There's a difference between putting information easily online for your customers or even people in general (eg as a hobby), and working in concert with scraping for greater visibility via search, and giving that work away, or at a cost, to companies who at best don't care and possibly may be competition, see themselves as replacing you or otherwise adversarial. The line is "I technically and able to do this" and "I…

> The line is "I technically and able to do this" and "I am engaging with a system in good faith".

This is where the line should be, always. But in practice this criterion is applied very selectively here on HN and elsewhere.

After all: What is ad blocking, other than direct subversion of the site owner's clear intention to make money from the viewer's attention?

Applying your criterion here gives a very simple conclusion: If you don't want to watch the ads, don't visit the site.

Right?

Post reply on HN