Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

171–180 of 228 posts

Re: End of an era for me: no more self-hosted git

#171

Post by greiskul »

forgotmypw17 wrote: post #168
kstrauser wrote: post #126

Earlier quoted context omitted.

Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of us…

There are many reasons to accommodate non-JS users beyond accommodating people who have intentionally disabled it, and most of them are in accessibility territory. Be careful with using percentages for your arguments, because this is not that different from saying that 99.99% of people don't need wheelchair access.

This used to be true, but now I don't think it is anymore. Modern frameworks and modern screen readers have no issue with acessibility.

Some survey from WebAIM found that 99.3% of screen reader users have JavaScript enabled.

So... are they really in accessibility territory still? Only people I still see complaining about Javascript being required are people that insist the web should just be static documents with hyperlinks like it was in the early 90s.

Can you find a modern source with valid reasons for accomodating non-JS users?

Re: End of an era for me: no more self-hosted git

#172

Post by mikepavone »

recursivecaveat wrote: post #46
krick wrote: post #44

So, what's up with these bots, why am I hearing about that so often lately? I mean, DDoS atacks aren't a new thing, and, honestly, this is pretty much the reason why Cloudflare even exists, but I'd expect OpenAI bots (or whatever this is now) to be a little bit easier to deal with, no? Like, simply having resonable aggressive fail2ban policy? Or do they really behave like a botnet, where each request comes from diffe…

I doubt it's OpenAI. Maaaybe somebody who sells to OpenAI, but probably not. I think they're big enough to do this mostly in-house and properly. Before AI only big players would want a scrape of the entire internet, they could write quality bots, cooperate, behave themselves, etc. Now every 3rd tier lab wants that data and a billion startups want to sell it, so it's a wild west of bad behavior and bad implementations…

As someone with a self-hosted Mercurial instance dealing with this, I will say that the big names (OpenAI included, but not exclusively them) generally at least use proper user-agents and respect robots.txt, but they are still needlessly aggressive compared to traditional search indexers.

There are also scrapers that are hiding behind normal browser user agents. When I looked at IP ranges, at least some of them seemed to be coming from data centers in China.

Re: End of an era for me: no more self-hosted git

#173

Post by chipsrafferty »

xorcist wrote: post #120
kstrauser wrote: post #93

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…

Not enough people opt out of using js for it to matter to anyone. If a page doesn't work because you have js disabled, get over it

Re: End of an era for me: no more self-hosted git

#174

Post by chipsrafferty »

shakna wrote: post #151
kstrauser wrote: post #126

Earlier quoted context omitted.

Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of us…

> JS is as fundamental to current design as CSS. I think this hits the crux of the trend fairly well. And is why I have so many workarounds to shitty JS in my user files. Because I can't see your CSS, either.

Yet you use CSS on your own website?

Re: End of an era for me: no more self-hosted git

#175

Post by shimman »

reppap wrote: post #125
recursivecaveat wrote: post #46

Earlier quoted context omitted.

I doubt it's OpenAI. Maaaybe somebody who sells to OpenAI, but probably not. I think they're big enough to do this mostly in-house and properly. Before AI only big players would want a scrape of the entire internet, they could write quality bots, cooperate, behave themselves, etc. Now every 3rd tier lab wants that data and a billion startups want to sell it, so it's a wild west of bad behavior and bad implementations…

Stop just making up excuses for these companies. Other comments on this story have showed the bots are using openai user agents and making requests from openai owned ip ranges.

It's like how we "can't" stop spam callers when telecoms know exactly who is calling who, they just don't want to implement any protocols that benefit society because they rather make money while fucking over everyone.

Re: End of an era for me: no more self-hosted git

#176

Post by shakna »

chipsrafferty wrote: post #174
shakna wrote: post #151

Earlier quoted context omitted.

> JS is as fundamental to current design as CSS. I think this hits the crux of the trend fairly well. And is why I have so many workarounds to shitty JS in my user files. Because I can't see your CSS, either.

Yet you use CSS on your own website?

Yup. I do. And JS, too.

Because neither are _required_ for anything. There is a well-specified data tree.

Progressive enhancement is not some sign of conflict in my reasoning. It is a demonstration of it.

Re: End of an era for me: no more self-hosted git

#177

Post by golem14 »

dirkc wrote: post #121
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I'm hazarding a guess that there are many AI startups that focus on building datasets with the aim to sell those datasets. Still doesn't make total sense, since doing it badly would only hurt them, but maybe they don't really care about the product / outcome, they're just capturing their bit of the AI goldrush?

You can always prune it down later (so the thinking goes, no doubt)

Re: End of an era for me: no more self-hosted git

#178

Post by Tharre »

everybodyknows wrote: post #149
Tharre wrote: post #91

Earlier quoted context omitted.

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI? You don't really need to guess, it's obvious from the access logs. I realize not everyone runs their own server, so here are a couple excerpts from mine to illustrate: - "meta-externalagent/1.1 + https://developers.facebook.com/docs/sharing/webmasters/craw... )" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cl…

Following your link above, https://openai.com/gptbot > ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. So, not AI training in this case, nor any other large-batch scraping, but rather inference-time Retrieval Augmented Generation, with the "retrieval" happening over the web?

Those would have the user agent "ChatGPT-User" though, and I barely see those. The majority comes from "GPTBot" like in my excerpt above, which makes it pretty clear that it's used for some sort of training:

"GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."

Re: End of an era for me: no more self-hosted git

#179

Post by PaulDavisThe1st »

reactordev wrote: post #150
PaulDavisThe1st wrote: post #143

Earlier quoted context omitted.

We used fail2ban to do rate limiting first. It wasn't adequate.

Ooof, maybe a write up is in order? An opinioned blog post? I'd love to know more.

As noted by others, the scrapers do not seem to respond to rate limiting. When you're being hit by 10-100k different IP's per hour and they don't respond to rate limiting, rate limiting isn't very effective.

Re: End of an era for me: no more self-hosted git

#180

Post by raw_anon_1111 »

kstrauser wrote: post #139
tuhgdetzhh wrote: post #137

Earlier quoted context omitted.

I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.

It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.

Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.
Post reply on HN