The eternal fight between bots and anti-bot systems. The difference now is that big companies themselves promote/offer bots, but they also don't like to be scraped and use captchas. What do we do now? Is it allowed to use automated tools to interact with any system? Is it allowed to scrape data? Are there any laws for this? If we do things manually it is ok, but not if we use a bot? Confusing (legal) times...
> Is it allowed to use automated tools to interact with any system? I'd hope so, because that's what we're doing right now. Your browser is automatically speaking HTTP for you so that you don't have to. Am I having a bit of a laugh? Maybe. But really, services should be user-agent agnostic. That's the whole "agent" part of User Agent and the founders of the Internet had incredible foresight to name it this way. > Is…
Taking your position to its logical conclusion implies that we shouldn't try to mitigate DDoSes either.
In many cases, what the other server voluntarily transmits has so far been based on the tacit assumption that a person, with person-level time and computational power, is doing the receiving. While in principle a machine could be doing it even in pre-LLM times, in practice many websites, including all the biggest ones, have implemented a wide range of approaches to try to curb machine access, starting with user agent checks and rate limits but by no means ending there.
The question is: Given the new landscape, where this assumption increasingly does not hold (because AI agents are increasingly able to simulate anything a person could do online), would those servers voluntarily transmit that data? In many cases, the answer is no.