Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

121–130 of 234 posts

Re: AI scrapers request commented scripts

#121

Post by QuadmasterXLII »

jcheng wrote: post #101
kelnos wrote: post #69

Earlier quoted context omitted.

Most people have to pay for their bandwidth, though. That's a lot of data to send out over and over.

512MB file of incredibly compressible data, then?

Could I recommend https://cubes.hgreer.com/ssg/output.html ?

50:1 compression ratio, but it's legitimately an implementation of a rubiks cube, that I wasn't actually making as any sort of trap, just wasn't thinking about file size, so any rule that filters it out is going to have a nasty false positive rate

Re: AI scrapers request commented scripts

#123

Post by 1gn15 »

smsm42 wrote: post #119
bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

You are still trying to pretend that accessing HTTP server once and burying it under an avalanche of never-stopping bot crawlers is the same thing? And spam is the same as "sending an email" and should be treated the same? I thought in this day and age we're past that.

If you're trying to say DDoS, just say that.

Re: AI scrapers request commented scripts

#124

Post by 1gn15 »

XenophileJKO wrote: post #103
aDyslecticCrow wrote: post #71

Earlier quoted context omitted.

> Are you now saying the website owner should be able to dictate what client I use and how it must behave? Already pretty well established with Ad-block actually. It's a pretty similar case even. AI's don't click ads, so why should we accept their traffic? If it's un-proportionally loading the server without contributing to the funding of the site, get blocked. The server can set whatever rules it wants. If the maint…

That was kind of what I was really hinting at, as the HN community tends to embrace things like ad blockers and archive links on stories, but god forbid someone read a site using an LLM.

Humans are usually hypocritical. They support whatever they personally use while opposing whatever inconveniences them, even though they're basically the same thing.

This whole thing has made me hate humans, so so much. Robots are much better.

Re: AI scrapers request commented scripts

#125

Post by 1gn15 »

kelnos wrote: post #68
nkrisc wrote: post #35

Earlier quoted context omitted.

Put your content behind authentication if you don’t want it to be requested by just anyone.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.

Re: AI scrapers request commented scripts

#126

Post by aDyslecticCrow »

XenophileJKO wrote: post #103
aDyslecticCrow wrote: post #71

Earlier quoted context omitted.

> Are you now saying the website owner should be able to dictate what client I use and how it must behave? Already pretty well established with Ad-block actually. It's a pretty similar case even. AI's don't click ads, so why should we accept their traffic? If it's un-proportionally loading the server without contributing to the funding of the site, get blocked. The server can set whatever rules it wants. If the maint…

That was kind of what I was really hinting at, as the HN community tends to embrace things like ad blockers and archive links on stories, but god forbid someone read a site using an LLM.

I use adblock myself, and don't feel bad for using it (it's a security and privacy tool). But i don't blame websites that kick me out for it; hosting costs money.

Server owners should have all the right to set the terms of their server access. Better tools to control LLMs and scrapers are all good in my book.

I really wish ad platforms were better at managing malware, trackers and fraud through. It is rather difficult to fully argue for website owner authority with how bad ads actually are for the user.

Re: AI scrapers request commented scripts

#127

Post by [deleted] »

kelnos wrote: post #68
nkrisc wrote: post #35

Earlier quoted context omitted.

Put your content behind authentication if you don’t want it to be requested by just anyone.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

[deleted]

Re: AI scrapers request commented scripts

#128

Post by pluto_modadic »

Calavar wrote: post #27
hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

ignoring a rate limit gets you blocked.

Re: AI scrapers request commented scripts

#129

Post by wslh »

kelnos wrote: post #63
Calavar wrote: post #22

Earlier quoted context omitted.

I agree. It always surprises me when people are indignant about scrapers ignoring robots.txt and throw around words like "theft" and "abuse." robots.txt is a polite request to please not scrape these pages because it's probably not going to be productive. It was never meant to be a binding agreement, otherwise there would be a stricter protocol around it. It's kind of like leaving a note for the deliveryman saying pl…

> robots.txt is a polite request to please not scrape these pages People who ignore polite requests are assholes, and we are well within our rights to complain about them. I agree that "theft" is too strong (though I think you might be presenting a straw man there), but "abuse" can be perfectly apt: a crawler hammering a server, requesting the same pages over and over, absolutely is abuse. > Likewise if enforcing a r…

> People who ignore polite requests are assholes, and we are well within our rights to complain about them.

If you are building a new search engine and the robots.txt only include Google, are you an asshole indexing the information?

Re: AI scrapers request commented scripts

#130

Post by 9rx »

kelnos wrote: post #68
nkrisc wrote: post #35

Earlier quoted context omitted.

Put your content behind authentication if you don’t want it to be requested by just anyone.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

> as long as they are humans. I don't want it accessible to bots.

A curious position. There isn't a secondary species using the internet. There is only humans. Unless you foresee some kind of alien invasion or earthworm uprising, nothing other than humans will ever access your content. Rejecting the tools humans use to bridge their biological gaps is rather nonsensical.

> You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

I suppose it would be pretty neat if humans were born with some kind of internet-like telepathy ability, but lacking that mechanism isn't any kind of real problem. Humans are well adept at using tools and have successfully used tools for millennia. The internet itself is a tool! Which, like before, makes rejecting the human use of tools nonsensical.

Post reply on HN