Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

11–20 of 234 posts

Re: AI scrapers request commented scripts

#11

Post by mostlysimilar »

woodrowbarlow wrote: post #10
mostlysimilar wrote: post #7

Earlier quoted context omitted.

The article mentions using this as a means of detecting bots, not as a complaint that it's abusive. EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.

the first few words of the article are: > Last Sunday I discovered some abusive bot behaviour [...]

> The robots.txt for the site in question forbids all crawlers, so they were either failing to check the policies expressed in that file, or ignoring them if they had.

Re: AI scrapers request commented scripts

#12

Post by foobarbecue »

woodrowbarlow wrote: post #10
mostlysimilar wrote: post #7

Earlier quoted context omitted.

The article mentions using this as a means of detecting bots, not as a complaint that it's abusive. EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.

the first few words of the article are: > Last Sunday I discovered some abusive bot behaviour [...]

Yeah but the abusive behavior is ignoring robots.txt and scraping to train AI. Following commented URLs was not the crime, just evidence inadvertently left behind.

Re: AI scrapers request commented scripts

#14

Post by OhMeadhbh »

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references.

Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

Re: AI scrapers request commented scripts

#16

Post by rightbyte »

latenightcoding wrote: post #13

when I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.

DOM navigation for fetching some data is for tryhards. Using a regex to grab the correct paragraph or div or whatever is fine and is more robust versus things moving around on the page.

Re: AI scrapers request commented scripts

#17

Post by tuwtuwtuwtuw »

OhMeadhbh wrote: post #14

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

Do you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search?

I doubt it.

Re: AI scrapers request commented scripts

#18

Post by chaps »

rightbyte wrote: post #16
latenightcoding wrote: post #13

when I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.

DOM navigation for fetching some data is for tryhards. Using a regex to grab the correct paragraph or div or whatever is fine and is more robust versus things moving around on the page.

Doing both is fine! Just, once you've figured out your regex and such, hardening/generalizing demands DOM iteration. It sucks but it is what is is.

Re: AI scrapers request commented scripts

#19

Post by horseradish7k »

rightbyte wrote: post #16
latenightcoding wrote: post #13

when I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.

DOM navigation for fetching some data is for tryhards. Using a regex to grab the correct paragraph or div or whatever is fine and is more robust versus things moving around on the page.

but not when crawling. you don't know the page format in advance - you don't even know what the page contains!

Re: AI scrapers request commented scripts

#20

Post by bakql »

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs.

"Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server.

Yes, I know about weev. That was a travesty.

Post reply on HN