311–320 of 516 posts
I think a lot of really smart people are letting themselves get taken for a ride by the web scraping thing. Unless the bot activity is legitimately hammering your site and causing issues (not saying this isn't happening in some cases), then this mostly amounts to an ideological game of capture the flag. The difference being that you'll never find their flag. The only thing you win by playing is lost time. The best wa…
I'm always a little surprised to see how many people take robots.txt seriously on HN. It's nice to see so many folks with good intentions. However, it's obviously not a real solution. It depends on people knowing about it, and adding the complexity of checking it to their crawler. Are there other more serious solutions? It seems like we've heard about "micropayments" and "a big merkle tree of real people" type soluti…
robots.txt isn't the law
But in general being an asshole is not a crime.
Earlier quoted context omitted.
Let's suppose I'm running a residential proxy. Of course my home IP address changes every day, so you'll end up blocking my entire ISP (a major one) or city (a major one) one by one. And what if I'm behind CGNAT? You will block my entire ISP or city all in one go, and get complaints from a lot of people.
If enough websites block the entire ISP / city in this way, *and* enough users get annoyed by being blocked and switch ISPs, then the ISPs will be motivated to stay in business and police their customers' traffic harder. Alas, the "enough users get annoyed by being blocked and switch ISPs" step will never happen. Most users only care about the big web properties, and those have the resources to absorb such crawler tr…
Earlier quoted context omitted.
My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment
every single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked) if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't re…
Background scanner noise on the internet is incredibly common, but the AI scraping is not at the same level. Wikipedia has published that their infrastructure costs have notably shot up since LLMs started scraping them. I've seen similar idiotic behavior on a small wiki I run; a single AI company took the data usage from "who gives a crap" to "this is approaching the point where I'm not willing to pay to keep this site up." Businesses can "just" pass the costs onto the customers (which is pretty shit at the end of the day,) but a lot of privately run and open source sites are now having to deal with side crap that isn't relevant to their focus.
The botnets and DDOS groups that are doing mass scanning and testing are targeted by law enforcement and eventually (hopefully) taken down, because what they're doing is acknowledged as bad.
AI companies, however, are trying to make a profit off of this bad behavior and we're expected to be okay with it? At some point impacting my services with your business behavior goes from "it's just the internet being the internet" to willfully malicious.
Earlier quoted context omitted.
If enough websites block the entire ISP / city in this way, *and* enough users get annoyed by being blocked and switch ISPs, then the ISPs will be motivated to stay in business and police their customers' traffic harder. Alas, the "enough users get annoyed by being blocked and switch ISPs" step will never happen. Most users only care about the big web properties, and those have the resources to absorb such crawler tr…
The hapless end user won't blame the ISP first.
But my main point was in the second paragraph, that "enough of them would" will never happen anyway when the only ones doing the blocking are small websites.
Earlier quoted context omitted.
Zip bomb?
Doesn’t that tie up a socket on the server similarly to how a keepalive would on the bot user end?
Earlier quoted context omitted.
My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment
Thousands of requests per hour? So, something like 1-3 per second? If this is actually impacting perceived QoS then I think a gitea bug report would be justified. Clearly there's been some kind of a performance regression. Just looking at the logs seems to be an infohazard for many people. I don't see why you'd want to inspect the septic tanks of the internet unless absolutely necessary.
I've been working on a web crawler and have been trying to make it as friendly as possible. Strictly checking robots.txt, crawling slowly, clear identification in the User Agent string, single IP source address. But I've noticed some anti-bot tricks getting applied to the robot.txt file itself. The latest was a slow loris approach where it takes forever for robots.txt to download. I accidentally treated this as a 404…
Gating robots.txt might be a mistake, but it also might be a quick way to deal with crawlers who mine robots.txt for pages that are more interesting. It's also a page that's never visited by humans. So if you make it a tarpit, you both refuse to give the bot more information and slow it down.
It's crap that it's affecting your work, but a website owner isn't likely to care about the distinction when they're pissed off at having to deal with bad actors that they should never have to care about.
Earlier quoted context omitted.
> thousounds of requests an hour from bots That's not much for any modern server so I genuinely don't understand the frustration. I'm pretty certain gitea should be able to handle thousands of read requests per minute (not per hour) without even breaking a sweat.
Serving file content/diff requests from gitea/forgejo is quite expensive computationally. And these bots tend to tarpit themselves when they come across eg. a Linux repo mirror. https://social.hackerspace.pl/@q3k/114358881508370524
One time, sure. But unauthenticated requests would surely be cached, authenticated ones skip the cache (just like HN works :) ), as most internet-facing websites end up using this pattern.
Earlier quoted context omitted.
every single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked) if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't re…
I've heard this point raised elsewhere, and I think it's underplaying the magnitude of the issue. Background scanner noise on the internet is incredibly common, but the AI scraping is not at the same level. Wikipedia has published that their infrastructure costs have notably shot up since LLMs started scraping them. I've seen similar idiotic behavior on a small wiki I run; a single AI company took the data usage from…
but yeah the issue is that as long as you have something accessible to the public, it's ultimately your responsibility to deal with malicious/aggressive traffic
> At some point impacting my services with your business behavior goes from "it's just the internet being the internet" to willfully malicious.
I think maybe the current AI scraper traffic patterns are actually what "the internet being the internet" is from here forward