Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

91–100 of 228 posts

Re: End of an era for me: no more self-hosted git

#91

Post by Tharre »

data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI?

You don't really need to guess, it's obvious from the access logs. I realize not everyone runs their own server, so here are a couple excerpts from mine to illustrate:

- "meta-externalagent/1.1 +https://developers.facebook.com/docs/sharing/webmasters/craw...)"

- "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"

- "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36"

- "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"

- [...] (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"

And to give a sense of scale, my cgit instance recieved 37 212 377 requests over the last 60 days, >99% of which are bots. The access.log from nginx grew to 12 GiB in those 60 days. They scrape everything they can find, indiscriminately, including endpoints that have to do quite a bit of work, leading to a baseline 30-50% CPU utilization on that server right now.

Oh, and of course, almost nothing of what they are scraping actually changed in the last 60 days, it's literally just a pointless waste of compute and bandwidth. I'm actually surprised that the hosting companies haven't blocked all of them yet, this has to increase their energy bills substantially.

Some bots also seem better behaved then others, OpenAI alone accounts for 26 million of those 37 million requests.

Re: End of an era for me: no more self-hosted git

#92

Post by mzajc »

Imustaskforhelp wrote: post #87
mzajc wrote: post #68

Earlier quoted context omitted.

I'm also dealing with a scraper flood on a cgit instance. These conclusions come from just under 4M lines of logs collected in a 24h period. - Caching helps, but is nowhere near a complete solution. Of the 4M requests I've observed 1.5M unique paths, which still overloads my server. - Limiting request time might work, but is more likely to just cause issues for legitimate visitors. 5ms is not a lot for cgit, but with…

Quick question but do these bots which you mention are from a 24H period but how long will this "attack" continue for? Because this is something which is happening continuously & i have observed so many HN posts like these (Anubis iirc was created by its creator out of such frustration too). Git servers being scraped to the point of its effectively an DDOS.

Yes, the attack is continuous. The rate fluctuates a lot, even within a day. It's definitely an anomaly, because eg. from 2025-08-15 to 2025-10-05 I saw zero days with more than 10k requests. Here's a histogram of the past 2 weeks plus today.

  2026-01-28     21'460
  2026-01-29     27'770
  2026-01-30     53'886
  2026-01-31    100'114  #
  2026-02-01    132'460  #
  2026-02-02     73'933
  2026-02-03    540'176  #####
  2026-02-04    999'464  #########
  2026-02-05    134'144  #
  2026-02-06  1'432'538  ##############
  2026-02-07  3'864'825  ######################################
  2026-02-08  3'732'272  #####################################
  2026-02-09  2'088'240  ####################
  2026-02-10    573'111  #####
  2026-02-11  1'804'222  ##################

Re: End of an era for me: no more self-hosted git

#93

Post by kstrauser »

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h...

1. Anubis is a miracle.

2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t my idea; I link to the inspiration for it. I just included my Caddy-specific instructions for implementing it.)

Re: End of an era for me: no more self-hosted git

#94

Post by esseph »

Lerc wrote: post #66
esseph wrote: post #55

Earlier quoted context omitted.

Some fake user agent, some tell you who they are. Or.. do they? Here-in is the problem. And if you block them, you risk blocking actual customers.

If they are using appropriated hardware, what possible reason could there be for them saying who they are?

Three different "companies" normally:

1. The residential proxies

2. Scrapers, on behalf of or as an agent of the data buyer

3. Data buyer (ai training)

Scrapers are buying from residential proxies, giving the data buyer a bit of a shield/deniability.

The scrapers don't want to get outright blocked if they can avoid it, otherwise they have nothing to sell.

Re: End of an era for me: no more self-hosted git

#95

Post by octoberfranklin »

data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I don't think it has anything to do with LLMs.

I think the big cloud companies (AWS) figured out that they could scrape compute-intensive pages in order to drive up their customers' spend. Getting hammered? Upgrade to more-expensive instances. Not using cloud yet? We'll force you to.

The other possibility is cloudflare punishing anybody who isn't using it.

Probably a combination of these two things. Whoever's behind this has ungodly supplies of cheap bandwidth -- more than any AI company does. It's a cloud company.

Re: End of an era for me: no more self-hosted git

#96

Post by Imustaskforhelp »

devsda wrote: post #38
simonw wrote: post #25

Earlier quoted context omitted.

I would love to understand this. Just a few years ago badly behaved scrapers were rare enough not to be worth worrying about. Today they are such a menace that hooking any dynamic site up to a pay-to-scale hosting platform like Vercel or Cloud Run can trigger terrifying bills on very short notice. "It's for AI" feels like lazy reasoning for me... but what IS it for? One guess: maybe there's enough of a market now for…

For whatever reason, legislation is lax right now if you claim the purpose of scraping is for AI training even for copyrighted material. May be everyone is trying to take advantage of the situation before law eventually catches up.

> For whatever reason, legislation is lax right now if you claim the purpose of scraping is for AI training even for copyrighted material

I think the reason is that America & China for the most part are also in AI arms race combined with an AI bubble and neither side would wish to lose literally any percieved advantage to them no matter the cost on others.

Also there is an immense lobbying effort against senators who propose for a stricter AI regulation.

https://www.youtube.com/watch?v=DUfSl2fZ_E8 [What OpenAI doesn't want you to know]

It's actually a great watch. Highly recommended because a lot of talks about regulations does feel to me as mirrors and smoke.

Re: End of an era for me: no more self-hosted git

#97

Post by octoberfranklin »

M95D wrote: post #56
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I stopped trying to understand. Encountering a 404 on my site leads directly to a 1 year ban.

They're rotating through huge pools of residential IP addresses.

Re: End of an era for me: no more self-hosted git

#98

Post by Imustaskforhelp »

mzajc wrote: post #92
Imustaskforhelp wrote: post #87

Earlier quoted context omitted.

Quick question but do these bots which you mention are from a 24H period but how long will this "attack" continue for? Because this is something which is happening continuously & i have observed so many HN posts like these (Anubis iirc was created by its creator out of such frustration too). Git servers being scraped to the point of its effectively an DDOS.

Yes, the attack is continuous. The rate fluctuates a lot, even within a day. It's definitely an anomaly, because eg. from 2025-08-15 to 2025-10-05 I saw zero days with more than 10k requests. Here's a histogram of the past 2 weeks plus today. 2026-01-28 21'460 2026-01-29 27'770 2026-01-30 53'886 2026-01-31 100'114 # 2026-02-01 132'460 # 2026-02-02 73'933 2026-02-03 540'176 ##### 2026-02-04 999'464 ######### 2026-02-0…

Thanks for sharing the data, This unpredictability and everything is even more suspicious.

Thoughts on having an ssh server with https://github.com/charmbracelet/soft-serve instead?

Re: End of an era for me: no more self-hosted git

#99

Post by trillic »

rozab wrote: post #74
embedding-shape wrote: post #70

Earlier quoted context omitted.

I just threw up a public Forjego instance for some lightweight collaboration. About 2 minutes after the certificate was created, I'm guessing they picked up the instance from the transparency logs for certificates, and started going through every commit and so on from the two repositories I had added. Watched it for a while, thinking eventually it'd end. It didn't, seemed like Claudebot and GPTBot (which was the only…

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPTbot, ClaudeBot, PerplexityBot, and others. I added a robots.txt with explicit UAs for known scrapers (they seem to ignore wildcards), and after a few days the traffic died down completely and I've had no problem since. Git frontends are basically a tarpit so a…

I will second a good robots.txt. Just checked my metrics and < 100 requests total to my git instance in the last 48 hours. Completely public, most repos are behind a login but there are a couple that are public and linked.
Post reply on HN