Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

141–150 of 228 posts

Re: End of an era for me: no more self-hosted git

#141

Ugh, exposing it with cgit is why. Put it all behind an OAuth login using something like Keycloak and integrate that into something like GitLab, Forgejo, Gitea if you must. However. To host git, all you need is a user and ssh. You don’t need a web ui. You don’t need port 443 or 80.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over. We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Yuk…

   http {
       # ... other http settings
       limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
       # ...
   }


    server {
        # ... other server settings
        location / {
            limit_req zone=mylimit burst=20 nodelay;
            # ... proxy_pass or other location-specific settings
        }
    }

Rate limit read-only access at the very least. I know this is a hard problem for open source projects that have relied on web access like this for a while. Anubis?

Re: End of an era for me: no more self-hosted git

#142

This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )

Alternatively, from the nginx config file for git.ardour.org: location ~ commit/* { return 404; }

[deleted]

Re: End of an era for me: no more self-hosted git

#143

Earlier quoted context omitted.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over. We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Yuk… http { # ... other http settings limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; # ... } server { # ... other server settings location / { limit_req zone=mylimit burst=20 nodelay; # ... proxy_pass or other location-specific settings } } Rate limit read-only access at the very least. I know this is a hard problem for open source projects that have relied on web access like this for a while. Anubis…

We used fail2ban to do rate limiting first. It wasn't adequate.

Re: End of an era for me: no more self-hosted git

#144

Earlier quoted context omitted.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over. We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Every commit. Every diff between 2 different commits. Every diff with different query parameters. Git blame for each line of each commit. Imagine a task to enumerate every possible read-only command you could make against a Git repo, and then imagine a farm of scrapers running exactly one of them per IP address. Ugh.

Ugh Ugh Ugh ... and endless ughs, when all they needed was "git clone" to get the whole thing and spend as much time and energy as they wanted analyzing it.

Re: End of an era for me: no more self-hosted git

#145

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…

I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.

> Turns out careless, unethical vibecoders aren't very competent.

Well they are scraping web pages from a git forge, where they could just, you know, clone the repo(s) instead.

Re: End of an era for me: no more self-hosted git

#148
post #110

Earlier quoted context omitted.

Not so many... And there are tools to scan for dead links.

Can you scan my bookmarks? :) edit: i.e. if someone has a bookmark to a page on your site and it goes 404, then they are blocked for a year. You have no ability to scan it because it's a file on their local system.

Oh, now I understand.

I never removed anything, but I'll keep this in mind for the future.

Re: End of an era for me: no more self-hosted git

#149
post #91

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI? You don't really need to guess, it's obvious from the access logs. I realize not everyone runs their own server, so here are a couple excerpts from mine to illustrate: - "meta-externalagent/1.1 + https://developers.facebook.com/docs/sharing/webmasters/craw... )" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cl…

Following your link above, https://openai.com/gptbot

> ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.

So, not AI training in this case, nor any other large-batch scraping, but rather inference-time Retrieval Augmented Generation, with the "retrieval" happening over the web?

Re: End of an era for me: no more self-hosted git

#150

Earlier quoted context omitted.

Yuk… http { # ... other http settings limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; # ... } server { # ... other server settings location / { limit_req zone=mylimit burst=20 nodelay; # ... proxy_pass or other location-specific settings } } Rate limit read-only access at the very least. I know this is a hard problem for open source projects that have relied on web access like this for a while. Anubis…

We used fail2ban to do rate limiting first. It wasn't adequate.

Ooof, maybe a write up is in order? An opinioned blog post? I'd love to know more.
Post reply on HN