Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

knownagents.com

Post reply on HN
Print view View original

101–110 of 243 posts

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#101

Post by an0malous »

drewnick wrote: post #21
yabones wrote: post #6

Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.

Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.

Is that basically the delta between Cloudflare’s Pages analytics and Google Analytics? CF says I get thousands of visits a day and GA says it’s like 3-4 users.

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#102

Post by Bender »

VogonPoetry wrote: post #80
Bender wrote: post #73

Earlier quoted context omitted.

There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit bu…

Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6. Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.

I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well.

[1] - https://caniuse.com/?search=sec-fetch

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#103

Post by gavinhking »

an0malous wrote: post #101
drewnick wrote: post #21

Earlier quoted context omitted.

Think about how many webmaster and business owners' egos are stroked by all the traffic they are getting, when in actuality they are often just serving thousands of bots.

Is that basically the delta between Cloudflare’s Pages analytics and Google Analytics? CF says I get thousands of visits a day and GA says it’s like 3-4 users.

Basically, unless CF is counting static asset network requests etc. For what it's worth, GA also miscategorizes some bots as humans as well.

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#104

Post by [deleted] »

djkurlander wrote: post #15
codegeek wrote: post #12

Earlier quoted context omitted.

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.

I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net API: http://knock-knock.net/api

[deleted]

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#105

Post by itsTyrion »

binaryturtle wrote: post #91
heywire wrote: post #53

Earlier quoted context omitted.

This is one reason I don’t mind that I’m behind CGNAT.

That's the only advantage of CGNAT I can think of. :D (You still could get poked from the other users' hosts behind the ISP's NAT, of course.)

an angry skid in a game can't (D)DoS your router might be one. I'd still prefer to not have CGNAT

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#106

Post by cpburns2009 »

yabones wrote: post #6

Every server with port 80/443 open has thousands of hits a day from random boxes looking for wordpress login pages. The only new thing is that they're pretending to be a different type of annoying bot. There's a new layer of sophistication and subterfuge, but it's the same junk traffic we've always dealt with.

Exactly this. It's no different from a bot pretending to be Googlebot. I've tried reporting abusive IPs to various foreign hosts, but nothing every comes to it. I've settled for just blacklisting excessively abusive IP ranges.

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#107

Post by gavinhking »

reincoder wrote: post #79
gavinhking wrote: post #69

Earlier quoted context omitted.

Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?

You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides. If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first. I have quite a few community posts ar…

Appreciate it, I'll check out your posts.

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#108

Post by codegeek »

djkurlander wrote: post #15
codegeek wrote: post #12

Earlier quoted context omitted.

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.

I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net API: http://knock-knock.net/api

Thank you for sharing. I will take a look.

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#109

Post by Bender »

djkurlander wrote: post #15
codegeek wrote: post #12

Earlier quoted context omitted.

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.

I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net API: http://knock-knock.net/api

Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects.

[1] - https://adver.tools/reverse-dns-lookup/

Re: Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

#110

Post by red-iron-pine »

Tharre wrote: post #51

Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive. Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.

Googlebot and other web scrapers are how you get visible, how you get adwords, ways to boost your traffic, etc.

bots hitting your site aren't problems per se

Post reply on HN