251–260 of 291 posts
Why can't Cloudflare do it? They know in real time which IPs spew millions of scrape requests to various sites, so they can classify them as AI bots and allow site owners to block them.
Some people have issues with the idea that the entire web will eventually depend of a single point of failure and private for provide entity like cloudflare.
If you're worried about your data getting scraped and used then maybe you can consider putting it behind a login or do some proof of work/soft captcha. Yeah, this isn't perfect but it will keep most dumb bots away.
Some people are hosting their sites like we're still in 1995 and times have changed.
Earlier quoted context omitted.
Check out Anubis - it's not quite what you're suggesting but similar in concept: https://anubis.techaro.lol/
Maybe once you can use something more professional as an interstitial page
Earlier quoted context omitted.
You're saying that large companies are hitting individual websites with thousands of unrelated IP addresses?
How do you think they do crawling if not like that? They'd be IP banned instantly if they used any kind of predictable IP regime for more than a few minutes.
Also you're implying that the only way to crawl is to essentially DDOS a website by blasting them from thousands of IP addresses. There is no reason crawlers can't do more sites in parallel and avoid hitting individual sites so hard. There are plenty of crawlers for the last few decades that don't cause problems, these are just stories about the ones that do.
Earlier quoted context omitted.
At which level of DDos one can claim damages from them?
You can claim whatever you want, but actually litigation is expensive and it’s not at all a sure thing that “I made a publicly available resource and they used it too much” is going to win damages. Maybe? Maybe not?
Earlier quoted context omitted.
You might say that, but claiming everyone is a pedophile is such a tired political play at this point. Its primary purpose is to dehumanize people so that blatant wrongdoing can be justified. The visceral reaction might be genuine, but the actual feelings are probably not. I have yet to see someone who actually "cares about the children". The vast majority of accusations e.g. democrats running a pedophile ring turn o…
[flagged]
Earlier quoted context omitted.
Re-read what I posted and stop projecting.
But if the AI crawlers are taking the website down and money buys more server time, are you willing to do your part and use money to make sure your training data sources are solvent until you can replace them?
Sure, overly spamming websites is shitty behaviour. But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?
So you're willing to pay my hosting bills?