Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FOSS infrastructure is under attack by AI companies

thelibre.news

Post reply on HN
Print view View original

371–380 of 631 posts

Re: FOSS infrastructure is under attack by AI companies

#371

Post by rhaps0dy »

Vegenoid wrote: post #344
pca006132 wrote: post #260

Earlier quoted context omitted.

There are off-chain solutions to handle most of the payment, and only put a summary on-chain. I think there are already micropayments in Brave or something.

At this point, why even use bitcoin, or crypto at all. Just use a crypto that is better suited to micropayments.

Proof-of-work crypto is interesting here because it is fungible with computation, so these solutions that charge computation to users are literally equivalent to crypto.

It's a solution that already has adoption, does not require everyone to sign up with a centralized service, and does not require everyone to pay money (they can pay with small amounts of computation instead) so it remains accessible to ~everyone.

Re: FOSS infrastructure is under attack by AI companies

#372

Post by 999900000999 »

Aachen wrote: post #234
aspir wrote: post #34

Just a callout that Fastly provides free bot detection, CDN, and other security services for FOSS projects, and has been for 10+ years https://www.fastly.com/fast-forward (disclaimer, I work for Fastly and help with this program) Without going into too much detail, this tracks with the trends in inquiries we're getting from new programs and existing members. A few years ago, the requests were almost exclusively relat…

I've been running into bot detection on at least five different websites in the past two months (not even including captcha walls) Not sure what to tell you but I surely feel quite human Three of the pages told me to contact customer support and the other two were a hard and useless block wall. Only from Codeberg did I get a useful response, the other two customer supports were the typical "have you tried clearing yo…

To be fair,

>I'm Not a Robot (film) https://en.m.wikipedia.org/wiki/I%27m_Not_a_Robot_(film)

Re: FOSS infrastructure is under attack by AI companies

#374

Post by stego-tech »

> ...15% is due to Amazon

And you just know they'll gladly bill you for egress charges for their own bot traffic, too.

EDIT: Actually, this is an excellent question. By default, these bots would likely appear to come from "the internet" and thus be subject to egress charges for data transfers. Since all three major cloud providers also have significant interests in AI, wouldn't this be a sort of "silent" price increase, or a form of exploitive revenue pumping? There's nothing stopping Google, Microsoft/OpenAI, or Amazon from sending an army of bots against your sites, scraping the data, and then stiffing you with the charges for their own bots' traffic. Would be curious if anyone has read the T&Cs of their own rate cards closely enough to see if that's the case, or has proof in their billing metrics.

---

Original post continues below:

One topic of conversation I think worth having in light of this is why we still agree to charge for bandwidth consumed instead of bandwidth available, just as general industry practice. Bits are cheap in the grand scheme of things, even free, since all the associated costs are for the actual hardware infrastructure and human labor involved in setup and maintenance - the actual cost per bit transmitted is ridiculously small, infinitesimally so to be practical to bill.

It seems to me a better solution is to go back to charging for capacity instead of consumption, at least in an effort to reduce consumption charges for projects hosted. In the meantime, I'm 100% behind blocking entire ASNs and IP blocks from accessing websites or services in an effort to reduce abuse. I know a prior post about blocking the entirety of AWS ingress traffic got a high degree of skepticism and flack from the HN community about its utility, but now more than ever it seems highly relevant to those of us managing infrastructure.

Also, as an aside: all the more reason not to deploy SRV records for home-hosted services. I suspect these bots are just querying standard HTTP/S ports, and so my gut (but NOT data - I purposely don't collect analytics, even at home, so I have NO HARD EVIDENCE FOR THIS CLAIM) suggests that having nothing directly available on 80/443 will greatly limit potential scrapers.

Re: FOSS infrastructure is under attack by AI companies

#375

Post by Spivak »

UncleMeat wrote: post #335
dbingham wrote: post #296

Earlier quoted context omitted.

That value is only great if it's shared equitably with the rest of the planet. If it's owned by a few, as it is right now, it's an existential threat to the life, liberty, and pursuit of a happiness of everyone else on the planet. We should be seriously considering what we're going to do in response to that threat if something doesn't change soon.

Yep. The "wouldn't it be great if we had robots do all the labor you are currently doing" argument only works if there is some plan to make sure that my rent gets paid other than me performing labor.

It depends if you're the only one out of a job. If it really is everyone then the answer will likely be some variant of metaphorically or literally killing your landlord in favor of a different resource allocation scheme. I put these kinds of things in a "in that world I would have bigger problems" bucket.

Re: FOSS infrastructure is under attack by AI companies

#376

Post by TheRealPomax »

Cthulhu_ wrote: post #249

Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation…

How would legislation in the US or EU stop traffic from China or Thailand or Russia? At best you'd be fragmenting the internet, which isn't really a "best", that's a terrible idea.

Re: FOSS infrastructure is under attack by AI companies

#377

Post by reginald78 »

xena wrote: post #33

It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis . So far I've noticed that it seems to actually work. I just deployed it to xeiaso.net as a way to see how it fails in prod for my blog.

Maybe I'm missing something, but doesn't this mean the work has to be done by the client AND the server every time a challenge is issued? I think ideally you'd want work that was easy for the server and difficult for the server. And what is to stop being DDoS'd by clients that are challenged but neglect to perform the challenge?

Regardless, I think something like this is the way forward if one doesn't want to throw privacy entirely out the window.

client

Re: FOSS infrastructure is under attack by AI companies

#378

Post by TonyTrapp »

superkuh wrote: post #276

To me it sounds like these people are operating websites that don't work. My home internet (80 down, 5 up) connection hosted website handled 20k+ hits from the alibaba ai crawler yesterday without missing a beat. And many thousands more from GPTbot, etc. I'll grant it can be a problem for super-heavy "application" websites where every GET is a serious computation. So I'm not surprised gitlab is having problems. They'…

There's source repository browsers (git/svn) way, way leaner than GitLab that have the same issues. Any repo browser offering a blame view for files can be brought down by those bots' traffic patterns. I have been hosting such repository browsers for 10+ years and it was never an issue until the arrival of these bots.

Re: FOSS infrastructure is under attack by AI companies

#379

Post by knowaveragejoe »

xena wrote: post #33

It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis . So far I've noticed that it seems to actually work. I just deployed it to xeiaso.net as a way to see how it fails in prod for my blog.

I love that I seem to stumble upon something by you randomly every so often. I'd just like to say that I enjoy your approach to explanations in blog form and will look further into Anubis!

Re: FOSS infrastructure is under attack by AI companies

#380

Post by floren »

ozornin wrote: post #345
floren wrote: post #288

Earlier quoted context omitted.

Just change the pictures in cmd/anubis/static/img/ to whatever you prefer, I think.

Or, alternatively, you know, pay the author for the work they've done

Sure, if you're going to deploy it on your company site, but I think if you're running a personal website and want to throttle LLM crawlers without falsely advertising that you're a furry, you could just go and modify this piece of MIT-licensed software.
Post reply on HN