61–70 of 97 posts
Earlier quoted context omitted.
> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?
The point is that not every web crawler is out there to scrape websites.
> It seems quite likely that this is being done via a botnet - illegally abusing thousands of people's devices. Sigh. Just because traffic is coming from thousands of devices on residential IPs, doesn't mean it's a botnet in the classical sense. It could just as well be people signing up for a "free VPN service" — or a tool that "generates passive income" for them — where the actual cost of running the software, is t…
Easiest way to deal with them is just to block them regardless, because the probability that someone who knows what to do about this software and why it's bad will read any particularly botnetted website are close to zero.
This makes me wonder if there are more efficient image formats that one might want to feed botnets. JPEG is highly complex, but PNG uses a relatively simple DEFLATE stream as well as some basic filters. Perhaps one could make a zip-bomb like PNG that only consists of a few bytes?
I am bit confused about the context. What is exactly the point of exposing fake data to webcrawlers?
Is there reason you couldn’t generate your images by grabbing random rectangles of pixels from one source image and pasting it into a random location in another source image? Then you would have a fully valid jpg that no AI could easily successfully identify as generated junk. I guess that would require much more CPU than your current method huh?