Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

461–470 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#461

Post by danaris »

brushfoot wrote: post #39

At this rate, it's more than FOSS infrastructure -- although that's a canary in the coalmine I especially sympathize with -- it's anonymous Internet access altogether. Because you can put your site behind an auth wall, but these new bots can solve the captchas and imitate real users like never before. Particularly if they're hitting you from residential IPs and with fake user agents like the ones in the article -- or…

I dunno. I run a small browser-game, and while my server has been periodically getting absolutely pulverized by LLM scrapers, I have yet to see a single new account that looks remotely like it was created by a bot. (Also, the rate of new signups hasn't changed notably.) This is true for both the game and its Wiki—which is where most of the scraping traffic has been. (And which I will almost certainly have to set to be almost-completely authwalled if the scraping doesn't let up.)

Re: FOSS infrastructure is under attack by AI companies

#462

Post by MattGrommes »

I wonder if there's a way for a human-controller browser to generate some kind of token that proves it's being used by a human?

Obviously there's still ways to pay people to run the browser but it would be nice for this activity to cost the AI company something without blocking actual people.

Re: FOSS infrastructure is under attack by AI companies

#463

Post by ethan_smith »

lelanthran wrote: post #42

The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…

What about the next-gen of AI that would be able to signup autonomously? Even if implemented auth-walls everywhere right now, whats stopping the companies to get some real cheap labor to create accounts on websites and use them to scrape the content?

Is it going to become another race like the adblocker -> detect adblocker -> bypass adblocker detector and so on...?

Re: FOSS infrastructure is under attack by AI companies

#464

Post by danaris »

blueflow wrote: post #133
nzeid wrote: post #57

We need a project in the spirit of Spamhaus to actively maintain a list of perpetrating IPs. If they're cycling through IPs and IP blocks I don't know how sustainable a CAPTCHA-like solution is.

Why only in the "spirit of Spamhaus"? Spamhaus still exists. Add Google and Microsoft AS to the DROP/NOROUTE list, that would be hilarious.

Because while this is clearly related to spam, it's not the same thing, and presumably if Spamhaus themselves felt it was within their wheelhouse, they'd already be doing it.

Re: FOSS infrastructure is under attack by AI companies

#465

Post by ethan_smith »

RobKohr wrote: post #382
xyzal wrote: post #159

In case anyone is interested in a tiny bit of sabotage, I am under the impression I managed to 'drown' true information on my microblog by generating contradicting posts with LLaMa (tens of them for each real post) and invisibly linking them, so a human would not click through. You know, flood the zone with s***, Bannon-style ...

Yep, just make sure to add them to your robots.txt file so that only the bad robots are harmed.

Temporary solution and only works if only some of us are doing it. What if these bots have a "manager" LLM agent that takes the decision on what pages to scrape?

Re: FOSS infrastructure is under attack by AI companies

#466

Post by danaris »

QuadrupleA wrote: post #437
mrweasel wrote: post #120

Earlier quoted context omitted.

From reading Drew Devaults angry post from earlier this week, my take is that not only is it poorly implemented crawlers, it's also that it's cheaper to scrape, rather than keep copies on hand. Effectively these companies are outsourcing the storage of "their" training data to everyone on the internet. Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while als…

That seems crazy - millions of $ on GPUs but they can't afford some cheap storage? And direct network scraping seems super high latency. Although I guess a massive pretaining run might cycle through the corpus very slowly. Dunno, sounds fishy.

It's not that they can't afford storage.

It's that not doing so means they can increase their profit numbers just a skoshe more.

And at least as long as they haven't IPOed, that number's the only thing that matters. Everything getting in the way of increasing it is just an obstacle to be removed.

Re: FOSS infrastructure is under attack by AI companies

#467

Post by elaus »

Suppafly wrote: post #451
dreamcompiler wrote: post #413

Earlier quoted context omitted.

> It's trivial to setup a basic account for any site that doesn't contain any personal data you want to keep hidden It's equally trivial for a restaurant to set up a custom domain with their own 2-page website (overview and menu) on any of a hundred platforms that provide this service. Most of these services are not free like FB, but any business that can afford a landline phone can afford a real website.

>It's equally trivial for a restaurant to set up a custom domain with their own 2-page website (overview and menu) on any of a hundred platforms that provide this service. Sure but they don't want to. If you want to see the menu they have online you need to follow their rules, not your own.

And if they want me as a customer, they have to follow _my_ rules.

Obviously the restaurant has enough other customers and I have enough other restaurants to go to, so we both will be fine.

Re: FOSS infrastructure is under attack by AI companies

#468

Post by Freebytes »

ericholscher wrote: post #3

Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…

Along with having block lists, perhaps you could add poison to your results that generates random bad code that will not work, and that is only seen by bots (display: none when rendered), and the bots will use it, but a human never would.

Re: FOSS infrastructure is under attack by AI companies

#469

Post by ethan_smith »

Snowfield9571 wrote: post #449

I wonder how free hosting services like netlify and vercel handle this? If a free tier user gets spammed do they just pass the cost on? I can't imagine that is the case so they must have some protections built in.

That's bold of you to assume. Vercel has horror stories even for the paid tier.

Re: FOSS infrastructure is under attack by AI companies

#470

Post by j2kun »

TheRealPomax wrote: post #376
Cthulhu_ wrote: post #249

Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation…

How would legislation in the US or EU stop traffic from China or Thailand or Russia? At best you'd be fragmenting the internet, which isn't really a "best", that's a terrible idea.

This is the key point, but if US laws are being violated and AI is considered part of national security, that could be used by the US government in international negotiations, and for justification for sanctions, etc. It would be a good deterrent.
Post reply on HN