Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

131–140 of 228 posts

Re: End of an era for me: no more self-hosted git

#131

Post by karel-3d »

esseph wrote: post #50
krick wrote: post #44

So, what's up with these bots, why am I hearing about that so often lately? I mean, DDoS atacks aren't a new thing, and, honestly, this is pretty much the reason why Cloudflare even exists, but I'd expect OpenAI bots (or whatever this is now) to be a little bit easier to deal with, no? Like, simply having resonable aggressive fail2ban policy? Or do they really behave like a botnet, where each request comes from diffe…

The dirty secret is a lot of them come through "residential proxies", aka backdoored home routers, iot devices with shitty security, etc. Basically the scrapers who are often also third party, go to these "companies" and buy access to these "residential proxies". Some are more... considerate than others. Why? Data. Every bit of it is it might be valuable. And not to sound tin foil hatty, but we are getting closer to…

it isn't that hard to just buy a bunch of sim cards and put them in a modem and use that. it's good enough as a residential proxy. source: I did that before, when I worked on plaid-like thing.

Re: End of an era for me: no more self-hosted git

#132

Post by PaulDavisThe1st »

moebrowne wrote: post #83

This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )

Alternatively, from the nginx config file for git.ardour.org:

   location ~ commit/* {
        return 404;
    }

Re: End of an era for me: no more self-hosted git

#133

Post by PaulDavisThe1st »

reactordev wrote: post #123

Ugh, exposing it with cgit is why. Put it all behind an OAuth login using something like Keycloak and integrate that into something like GitLab, Forgejo, Gitea if you must. However. To host git, all you need is a user and ssh. You don’t need a web ui. You don’t need port 443 or 80.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over.

We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Re: End of an era for me: no more self-hosted git

#134

Post by thesuitonym »

tigerlily wrote: post #63
esseph wrote: post #50

Earlier quoted context omitted.

The dirty secret is a lot of them come through "residential proxies", aka backdoored home routers, iot devices with shitty security, etc. Basically the scrapers who are often also third party, go to these "companies" and buy access to these "residential proxies". Some are more... considerate than others. Why? Data. Every bit of it is it might be valuable. And not to sound tin foil hatty, but we are getting closer to…

How can I detect if my router is backdoored, or being used as a residential proxy?

The most surefire way would be to put a device between your router and your ONT/modem to capture the packets and see what requests are being sent. It's not complicated but it IS a lot of information to sift through.

Your router may have the ability to log requests, but many don't, and even if yours does, if you're concerned the device may be compromised, how can you trust the logs?

BUT, with all that said, these attacks are typically not very sophisticated. Most of the time they're searching for routers at 192.168.1.1 with admin/admin as the login credentials. If you have anything else set, you're probably good from 97% of attackers (This number is entirely made up, but seriously that percentage is high). You can also check for security advisories on your model of router. If you find anything that allows remote access, assume you're compromised.

---

As a final note, it's more likely these days that the devices running these bots are IoT devices and web browsers with malicious javascript running.

Re: End of an era for me: no more self-hosted git

#135

Post by GuestFAUniverse »

Fail2ban has decent jails for Apache httpd. And writing a rule that matches requests to nonexistent resources is very easy -- one-liners + time based threshold. Basically you could ban differently according to the http errors they cause (e.g. bots on migrated resources: many 404 within a minute, Slowloris is visible as a lot of 408).

Re: End of an era for me: no more self-hosted git

#136

Post by GuestFAUniverse »

GuestFAUniverse wrote: post #135

Fail2ban has decent jails for Apache httpd. And writing a rule that matches requests to nonexistent resources is very easy -- one-liners + time based threshold. Basically you could ban differently according to the http errors they cause (e.g. bots on migrated resources: many 404 within a minute, Slowloris is visible as a lot of 408).

Jails for other web servers are obviously as easy.

Re: End of an era for me: no more self-hosted git

#137

Post by tuhgdetzhh »

QuiDortDine wrote: post #108
kstrauser wrote: post #93

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…

I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.

I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.

Re: End of an era for me: no more self-hosted git

#139

Post by kstrauser »

tuhgdetzhh wrote: post #137
QuiDortDine wrote: post #108

Earlier quoted context omitted.

I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.

I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.

It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.

Re: End of an era for me: no more self-hosted git

#140

Post by kstrauser »

PaulDavisThe1st wrote: post #133
reactordev wrote: post #123

Ugh, exposing it with cgit is why. Put it all behind an OAuth login using something like Keycloak and integrate that into something like GitLab, Forgejo, Gitea if you must. However. To host git, all you need is a user and ssh. You don’t need a web ui. You don’t need port 443 or 80.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over. We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Every commit. Every diff between 2 different commits. Every diff with different query parameters. Git blame for each line of each commit.

Imagine a task to enumerate every possible read-only command you could make against a Git repo, and then imagine a farm of scrapers running exactly one of them per IP address.

Ugh.

Post reply on HN