I wonder how the internet would have been different if claws had existed beforehand. I keep thinking something simpler like Gopher (an early 90's web protocol) might have been sufficient / optimal, with little need to evolve into HTML or REST since the agents might be better able to navigate step-by-step menus and questionnaires, rather than RPCs meant to support GUIs and apps, especially for LLMs with smaller contex…
That's literally not possible would be my take. But of course just intuition.
The dataset used to train LLM:s was scraped from an internet. The data was there mainly due to the user expansion due to www, and the telco infra laid during and after dot-com boom that enabled said users to access web in the first place.
The data labeling which underpins the actual training, done by masses of labour, on websites, could not have been scaled as massively and cheaply without www scaled globally with affordable telecoms infra.