61–70 of 758 posts
Earlier quoted context omitted.
Apparently, the regular search crawler does it, but the ai thingie doesn't.
Huh? You can add Google-Extended[1] to opt out from Generative AI summaries. [1] https://blog.google/technology/ai/an-update-on-web-publisher...
I wonder if it will actually respect the robots.txt this time.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
I wonder if it will actually respect the robots.txt this time.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
So, similarly, LLM companies can see this as a signal to crawl to whole site to add to their training sets and learn from it, if the same URL is hit for a couple of times in a relatively short time period.
Earlier quoted context omitted.
It must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.
I remember in late 90s/early 2000 as a teen going to robots.txt to specifically see what they were trying to hide and exploring those urls. What is the difference if I use a browser or a LLM tool (or curl, or wget, etc) to make those requests?
I wonder if it will actually respect the robots.txt this time.
I don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
Earlier quoted context omitted.
Every automated crawler follows human-driven actions.
Conversely, every browser is a program that automatically executes HTTP requests.
So they sometimes hit bollards and turnstiles made for other types of code which executes HTTP requests. So they're bots basically, but better (or suitably) behaving ones.