Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

471–480 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#471

Post by scoofy »

God... I literally just coded up three different honeypots for this exact problem on my site, https://golfcourse.wiki, because LLM scrapers are a constant problem. I added a stupid recaptcha to the sign up form after literally 10,000 fake users were created by bots, averaging about 50 per day, and I have to say, recaptcha was suprisingly cumbersome to set up.

It's awful and it was costing me non-trivial amounts of money just from the constant pinging at all hours, for thousands of pages that absolutely do not need to be scraped. Which is just insane, because I actively design robots.txt to direct the robots to the correct pages to scrape.

So far so good with the honeypots, but I'll probably be creating more and clamping down harder on robots.txt to simply whitelist instead of blacklist. I'm thinking of even throwing in a robots honeypot directly in sitemap.xml that should bait robots to visit when they're not following the robots.txt.

It's really, really ridiculous.

Re: FOSS infrastructure is under attack by AI companies

#472

Post by Eridrus »

lostpencil wrote: post #455
Eridrus wrote: post #238

Earlier quoted context omitted.

I'm mostly suggesting a mirror to some centralized location. GitHub is probably a good place to mirror to for code, but it could be elsewhere. I was actually thinking of a more general thing than just code, eg similar to CommonCrawl, but maybe a code specific thing is what is needed.

Isn't this still similar to extortion? Maintainers aren't creating the problem. They are minding their own business until scrapers come along and make too many unnecessary requests. Seems like the burden is clearly on the scrapers. They could easily be hitting the pages much less often for a start. Doesn't your suggestion shift the responsibility to likely under-sponsored FOSS maintainers rather than companies? Also,…

Scrapers still have an economic incentive to do what is easiest. Providing an alternative that is easier than fighting sysadmin blocks would likely cause them to take the easier route and make it less of a cat and mouse game for sysadmins.

Re: FOSS infrastructure is under attack by AI companies

#473

Post by Eridrus »

gyesxnuibh wrote: post #303
Eridrus wrote: post #131

I know a lot of FOSS people are hostile to AI in general, and this is an immediate problem, but I feel like a better solution for everyone would be for there to be some sort of central repo of this information that AI companies can pull from without externalizing their costs like this.

Why buy the cow when you get the milk for free tho?

Because sysadmins are taking separate actions to make the milk expensive.

Re: FOSS infrastructure is under attack by AI companies

#474

Post by tremon »

tecleandor wrote: post #182
Galanwe wrote: post #53

Earlier quoted context omitted.

There's already loads of these. The problem is that most of these IPs are just cloud providers or DC ISPs.

Or even worse, lots of them are using barely legal residential proxies so the requests are coming from everywhere. In Drew DeVault's article linked in this post he complained precisely about the residential-looking source IP addresses [0]. And I think I remember something about a Chinese company, some months ago, very aggressively scraping using that method. Companies like DataImpulse [1] or ScraperAPI [2] will happi…

Are these "residential proxies" assumed to be infected devices part of a botnet or malicious apps on users' phones?

Re: FOSS infrastructure is under attack by AI companies

#475

Post by Y_Y »

djha-skin wrote: post #73
lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

I actually envision Liapunov stability, like wolf and rabbit populations. In this scenario, we're the rabbits. Human content will increase when AI populations decrease, this providing more food for AI, which will then increase. This drowns out human expression, and the humans will grow quieter. This provides less fodder for the AI, and they decrease. This means less noise and the humans grow louder. The cycle repeats…

See e.g. https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...

Re: FOSS infrastructure is under attack by AI companies

#476

Post by leerob »

Snowfield9571 wrote: post #449

I wonder how free hosting services like netlify and vercel handle this? If a free tier user gets spammed do they just pass the cost on? I can't imagine that is the case so they must have some protections built in.

If a free tier user goes over the included usage on Vercel, their site gets automatically paused. There isn't a credit card required to be put down.

Re: FOSS infrastructure is under attack by AI companies

#477

Post by skyyler »

diggan wrote: post #427
chneu wrote: post #304

Earlier quoted context omitted.

I'd be shocked if a single member of the US House or Congress know what robots.txt is.

I'm pleased to declare you shocked: https://www.google.com/search?q=site%3Acongress.gov+%22robot...

I don't know if "robots.txt" appearing in congressional record really counts. Do any of the decision makers appear to have a command of what the file does? Or do they typically relegate to industry professionals, as they often do?

Re: FOSS infrastructure is under attack by AI companies

#478

Post by programd »

enrico204 wrote: post #436
eb0la wrote: post #339

Earlier quoted context omitted.

If that happens, count with me to use Anubis to factor large primes or whatever science needs as a background task.

Actually, that is not a bad idea. @xena maybe Anubis v2 could make the client participate in some sort of SETI@HOME project, creating the biggest distributed cluster ever created :-D

Oh come now, clearly Anubis should make the clients mine bitcoin as proof of work, with a split for the website and the author.

Oh dear, somebody is going to implement this in about an hour, aren't they....

Re: FOSS infrastructure is under attack by AI companies

#480

Post by danaris »

Eridrus wrote: post #472
lostpencil wrote: post #455

Earlier quoted context omitted.

Isn't this still similar to extortion? Maintainers aren't creating the problem. They are minding their own business until scrapers come along and make too many unnecessary requests. Seems like the burden is clearly on the scrapers. They could easily be hitting the pages much less often for a start. Doesn't your suggestion shift the responsibility to likely under-sponsored FOSS maintainers rather than companies? Also,…

Scrapers still have an economic incentive to do what is easiest. Providing an alternative that is easier than fighting sysadmin blocks would likely cause them to take the easier route and make it less of a cat and mouse game for sysadmins.

Scrapers have an incentive to take all data.

If you put some data in a central repository, they will take it.

Then they will go and DDoS the rest of the Internet in order to take all the rest of the data.

Post reply on HN