AI scrapers request commented scripts
cryptography.dog
AI scrapers request commented scripts
1–10 of 234 posts
Re: AI scrapers request commented scripts
#2I'm not overly surprised, it's probably faster to search the text for http/https than parse the DOM
Re: AI scrapers request commented scripts
#3I'm not overly surprised, it's probably faster to search the text for http/https than parse the DOM
Not probably, searching through plaintext (which they seem to be doing) VS iterating on the DOM have vastly different amount of work behind them in terms of resources used and performance that "probably" is way underselling the difference :)
Re: AI scrapers request commented scripts
#4It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
Re: AI scrapers request commented scripts
#5It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
Re: AI scrapers request commented scripts
#6[deleted]
Re: AI scrapers request commented scripts
#7It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
The article mentions using this as a means of detecting bots, not as a complaint that it's abusive.
EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.
Re: AI scrapers request commented scripts
#8It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
The article mentions using this as a means of detecting bots, not as a complaint that it's abusive. EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".[1]
Re: AI scrapers request commented scripts
#9It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
Human behavior is interesting - me, me, me…
Re: AI scrapers request commented scripts
#10It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".
The article mentions using this as a means of detecting bots, not as a complaint that it's abusive. EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.
the first few words of the article are:
> Last Sunday I discovered some abusive bot behaviour [...]