Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

theregister.com

Post reply on HN
Print view View original

61–70 of 148 posts

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#61

Post by bwb »

skydhash wrote: post #58
bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

If you're not updating the publicly accessible part of the database open, try to see if you can put some cache strategy up and let cloudflare take the hit.

Yep, all but one page type is heavily cached at multiple levels. We are working to get the rest and improve it further... just annoying as they don't even respect limits..

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#62

Post by danny_codes »

prasadjoglekar wrote: post #5
breakyerself wrote: post #2

There's so much bullshit on the internet how do they make sure they're not training on nonsense?

By paying a pretty penny for non bullshit data (Scale Ai). That along with Nvidia are the shovels in this gold rush.

Making a lot of assumptions about the quality of scale AI.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#63

Post by mostlysimilar »

rnhmjoj wrote: post #54
pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.

But it didn't, and it's happening now, because of AI.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#64

Post by weaksauce »

bwb wrote: post #47

My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.

put a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#65

Post by jgalt212 »

about 18 months ago, our non-Google / Bing bot traffic went from single digits per cent to over 99.9% bot traffic. We tried some home-spun solutions at first, but eventually threw in the towel and put Cloudflare in front of all our publicly accessible pages. On a long term basis, this was probably the right move for us, but we felt forced into this. And the Cloudflare Managed Ruleset definitely blocks some legit traffic such that it requires a fair amount of manual tuning.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#66

Post by Group_B »

mcpar-land wrote: post #53

My worst offender for scraping one of my sites was Anthropic. I deployed an ai tar pit ( https://news.ycombinator.com/item?id=42725147 ) to see what it would do it with it, and Anthropic's crawler kept scraping it for weeks. I calculated the logs and I think I wasted nearly a year of their time in total, because they were crawling in parallel. Other scrapers weren't so persistent.

That's hilarious. I need to set up one of these myself

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#67

Post by superkuh »

pjc50 wrote: post #49

Place alongside https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher…

This isn't AI damaging anything. This is corporations damaging things. Same as it ever was. No need for scifi non-human persons when legal corporate persons exist. They latch on to whatever big new thing in tech that people don't understand which comes along and brand themselves with it and cause damage trying to make money; even if they mostly fail at it. And for most actual humans they only ever see or interact with the scammy corporation versions of $techthing and so come to believe $techthing = corporate behavior.

And as for denying service and preventing human people from visiting websites: cloudflare does more of that damage in a single day than all these "AI" associated corporations and their crappy crawlers have in years.

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#68

Post by buttercraft »

EgregiousCube wrote: post #42
xrd wrote: post #28

Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites. Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?

Under what law? It's interesting because these are sites that host content for the purpose of providing it to anonymous network users. ebay won a case against a scraper back in 2000 by claiming that the server load was harming them, but that reasoning was later overturned because it's difficult to say that server load is actual harm. ebay was in the same condition before and after a scrape. Maybe some civil lawsuit a…

If you put measures in place to prevent someone from accessing a computer, and they circumvent those measures, is that not a criminal offense in some jurisdictions?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#69

Post by recallingmemory »

okasaki wrote: post #52

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either coopera…

Yeah, I am in the opposing camp too - I don't use Cloudflare's bot fight tooling on any of our high traffic websites. I'm not seeing the issue with allowing bots to crawl our websites other than some additional spend for bandwidth. Agent mode is pretty powerful when paired with a website that cooperates, and if people want to use AI to interact with our data then what's wrong with that?

Re: AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

#70

Post by masfuerte »

okasaki wrote: post #52

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either coopera…

A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots.

The bot operators DNGAF.

Post reply on HN