> The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).
No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.
> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.
LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.