AI scrapers request commented scripts • Hacker News Board

Re: AI scrapers request commented scripts

#231

Post by davsti4 » Mon, Nov 03, 2025, 6:38 PM UTC

ColinWright wrote: post #107

davsti4 wrote: post #32

Earlier quoted context omitted.

Its simple, and I'll quote myself - "robots.txt isn't the law".

Quoting Cervisia : > robots.txt. This is not the law In Germany, it is the law. § 44b UrhG says (translated): (1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations. (2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are n…

Its the law pertaining to copyright. https://www.gesetze-im-internet.de/englisch_urhg/englisch_ur... ... and that's only in Germany.

If what you're protecting with robots.txt isn't copyright-able, then you'll need to find another legal means.

Re: AI scrapers request commented scripts

#232

Post by smsm42 » Tue, Nov 04, 2025, 4:57 AM UTC

davsti4 wrote: post #230

smsm42 wrote: post #228

Earlier quoted context omitted.

"I will do anything that will not literally get me into jail" is a pretty low bar. Most decent people try to do better than that - and that's the only reason society still exists, because there's not enough cops to put all bad people into jail and never will be.

Exactly - its why laws are written. So, if its enough of a problem, then there should be a law stopping/regulating it. Are published TOSs enough to stop the owners of the bots?

TOS wouldn't be enough but laws wouldn't be enough either. We have thousands of pages of laws, and people violate them all the times, and very often with zero consequences.

Re: AI scrapers request commented scripts

#233

Post by bigbuppo » Tue, Nov 04, 2025, 9:27 PM UTC

davsti4 wrote: post #229

bigbuppo wrote: post #213

Earlier quoted context omitted.

Violating norms makes you an abusive jerk at best.

That's a slippery slope opinion. One person's norm can oppose another's norm. Who defines them? If my "norm" is to stop at a stop sign, and yours is to roll through it, are you the abusive jerk?

No means no. Only yes means yes. Stop means stop. Why are people in tech so confused about this? Agree, or ask again in three days.

Re: AI scrapers request commented scripts

#234

Post by OhMeadhbh » Fri, Nov 07, 2025, 8:44 PM UTC

tuwtuwtuwtuw wrote: post #17

OhMeadhbh wrote: post #14

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

Do you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search? I doubt it.

Sure. Ideally CS programs would be more than the trade schools they are today. When I was in school, the "CS Program" was a specialization in the math department. I was already in a Physics program when they created a "Computer Science & Engineering" degree. And I remember both the Math and CSE degrees being pretty decent. But that was back in the day when people took CS related courses because they were interested in computers, not because they wanted to get a super high paying job for a FANG company.

Maybe you're right. You can lead a horse to water, but you can't make it think deep thoughts about parsing.