181–190 of 520 posts
Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
Why would you need to?
If your inability to assemble basic HTML forces you to adopt enormous, bloated frameworks that require two full cores of a cpu to render your post…
… or if you think your online missives are a step in the road to content creator riches …
… then I suppose I see the problem.
Otherwise there’s no problem.
I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot. Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.
The website owner has rights too. Are you arguing they cannot choose to implement such gatekeeping to keep their site operating in a financially viable manner?
There are people behind those connection requests. I don't try to guess on my server who is a bot and who is not; I'll make mistakes and probably bias against people who use uncommon setups (those needing accessibility aids or using e.g. experimental software that improves some aspect like privacy or functionality)
Sure, I have rights as a website owner. I can take the whole thing offline; I can block every 5th request; I can allow each /16 block to make 1000 requests per day; I can accept requests only from clients that have a Firefox user agent string. So long as it's equally applied to everyone and it's not based on a prohibited category such as gender or religious conviction, I am free to decide on such cuts and I'd encourage everyone to apply a policy that they believe is fair
Cloudflare and its competitors, as far as I can tell, block arbitrary subgroups of people based on secret criteria. It does not appear to be applied fairly, such as allowing everyone to make the same number of requests per unit time. I'm probably bothered even more because I happen to be among the blocked subgroup regularly (but far from all the time, just little enough to feel the pain)
Earlier quoted context omitted.
Allowlist is arguably fitting for a list of things which are allowed.
It's called a whitelist. A perfectly good word that isn't racist and one that normal people are quite happy to use. As far as I can tell the allow/blocklist craze hasn't made it out of the software world.
It's weird that people will claim that "politics" have no place in software while insisting that there is one and only one term "normal" people should use because the politics of the people who object to it are bad and wrong.
Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
> Everyone loves the dream of a free for all and open web... But the reality is how can someone small protect their blog or content from AI training bots? Aren't these statements entirely in conflict? You either have a free for all open web or you don't. Blocking AI training bots is not free and open for all.
We can't have nice things because the powerful cannot be held accountable. The powerful are powerful due to their legal teams and money, and power is the ability to carve exceptions to rules.
Earlier quoted context omitted.
How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.
How is it available for everyone if the AI bots bring down your server?
Earlier quoted context omitted.
I think that was the point. Everyone loves the dream, but the reality is different.
How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.
Freedom, the word, while implies no boundaries, is always bound by ethics, mutual respect and "do no harm" principle. The moment you trip either one of these wires and break them, the mechanisms to counter it becomes active.
Then we cry "but, freedom?!". Freedom also contains the consequences of one's actions.
Freedom without consequences is tyranny of the powerful.
Everyone loves the dream of a free for all and open web. But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real... Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed d…
Here's an even greater video: https://www.youtube.com/watch?v=mAUpxN-EIgU&t=4m24s
Earlier quoted context omitted.
How so? If you don't want AI bots reading information on the web, you don't actually want a free and open web. The reality of an open web is that such information is free and available for anyone.
How is it available for everyone if the AI bots bring down your server?
Talented people that want to scrape or bot things are going to find ways to make that look human. If that comes in the form of tricking a physical iPhone by automatically driving the screen physically, so be it; many such cases already!
The techniques you need for preventing DDoS don't need to really differentiate that much between bots and people unless you're being distinctly targeted; Fail2Ban-style IP bans are still quite effective, and basic WAF functionality does a lot.
Earlier quoted context omitted.
Anubis doesn’t necessarily stop the most well funded actors. If anything we’ve seen the rise in complaints about it just annoying average users.
The actual response to which Anubis was created is seemingly a strange kind of DDOS attack that has been misattributed to LLMs, but is some kind of attacker that makes partial GET requests that are aborted soon after sending the request headers, mostly coming from residential proxies. (Yes, it doesn’t help that the author of Anubis also isn’t fully aware of the mechanics of the attack. In fact, there is no proper wri…
um, no? Where did you get this strange bit of info.
The original reports say nothing of that sort: https://news.ycombinator.com/item?id=42790252 ; and even original motivation for Anubis was Amazon AI crawler https://news.ycombinator.com/item?id=42750420
(I've seen more posts with the analysis, including one which showed an AI crawler which would identify properly, but once it hits the ratelimit, would switch to fake user agent from proxies.. but I cannot find it now)