Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

141–150 of 234 posts

Re: AI scrapers request commented scripts

#141

Post by Mars008 »

renegat0x0 wrote: post #99

Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me.…

Looks like it's time for in-browser scrappers. They will be indistinguishable from the servers side. With AI driver can pass even human tests.

Re: AI scrapers request commented scripts

#142

Post by hsbauauvhabzb »

ahtihn wrote: post #93
halJordan wrote: post #39

Earlier quoted context omitted.

No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.

If a company was sending hundreds of salesmen to knock at a door one after the other, I'm pretty sure they could successfully get sued for harassment.

Can’t Americans literally shoot each other for trespassing?

Re: AI scrapers request commented scripts

#143

Post by hsbauauvhabzb »

czscout wrote: post #59
yuliyp wrote: post #36

Earlier quoted context omitted.

Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.

And a no soliciting sign is no more cosmically binding than robots.txt. It's a request, not an enforceable command.

Tell me you work in an ethically bankrupt industry without telling me you work in an ethically bankrupt industry.

Re: AI scrapers request commented scripts

#144

Post by hsbauauvhabzb »

pluto_modadic wrote: post #128
Calavar wrote: post #27

Earlier quoted context omitted.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

ignoring a rate limit gets you blocked.

Scrapers actively bypass this by rotating IP addresses.

Re: AI scrapers request commented scripts

#145

Post by smsm42 »

1gn15 wrote: post #123
smsm42 wrote: post #119

Earlier quoted context omitted.

You are still trying to pretend that accessing HTTP server once and burying it under an avalanche of never-stopping bot crawlers is the same thing? And spam is the same as "sending an email" and should be treated the same? I thought in this day and age we're past that.

If you're trying to say DDoS, just say that.

DDoS is a very specific type of attack. To be abusive, you don't have to do exactly that - it could be any type of DoS, and in fact it doesn't even have to deny all service - it could just impose excessive costs, for example.

Re: AI scrapers request commented scripts

#146

Post by dragonwriter »

hsbauauvhabzb wrote: post #142
ahtihn wrote: post #93

Earlier quoted context omitted.

If a company was sending hundreds of salesmen to knock at a door one after the other, I'm pretty sure they could successfully get sued for harassment.

Can’t Americans literally shoot each other for trespassing?

Generally, legally, no, not just for ignoring a “no soliciting” sign.

Re: AI scrapers request commented scripts

#147

Post by [deleted] »

jack_pp wrote: post #83
Larrikin wrote: post #60

Earlier quoted context omitted.

[flagged]

if u absolutely want a sexual metaphor it's more like you snuck into the world record for how many sexual parteners a woman can take in 24h and even tho you aren't on the list you still got to smash. solution is the same, implement better security

[deleted]

Re: AI scrapers request commented scripts

#148

Post by bigbuppo »

jack_pp wrote: post #83
Larrikin wrote: post #60

Earlier quoted context omitted.

[flagged]

if u absolutely want a sexual metaphor it's more like you snuck into the world record for how many sexual parteners a woman can take in 24h and even tho you aren't on the list you still got to smash. solution is the same, implement better security

Thank you for finding the right metaphor. If there is a sign out front that has a list of individuals that should go away but they continue, they're in a lot of legal trouble. If they show a fake ID to the event organizers that are handling all the paperwork, that is also something that will land them in prison.

Re: AI scrapers request commented scripts

#149

Post by jack_pp »

jraph wrote: post #90
jack_pp wrote: post #81

Earlier quoted context omitted.

here's my analogy, it's like you own a museum and you require entrance by "secret" password (your user agent filtering or what not). the problem is the password is the same for everyone so would you be surprised when someone figures it out or gets it from a friend and they visit your museum? Either require a fee (processing power, captcha etc) or make a private password (auth) It is inherently a cat and mouse game th…

> Either require a fee (processing power, captcha etc) or make a private password (auth) Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us. > It is inherently a cat and mouse game that you CHOOSE to play No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it…

That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me

Re: AI scrapers request commented scripts

#150

Post by kelseyfrog »

bigbuppo wrote: post #140
kelseyfrog wrote: post #135

Earlier quoted context omitted.

That's leaving a lot of opportunity on the table. The real money is in monetizing ad responses to AI scrapers so that LLMs are biased toward recommending certain products. The stealth startup I've founded does exactly this. Ad-poisoning-as-a-service is a huge untapped market.

Now that's a paid subscription I can get behind, especially if it suggests that Meta should cut Rob Schneider a check for $200,000,000,000 to make more movies.

Contact info in bio. Always looking to make more happy customers.
Post reply on HN