Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

71–80 of 228 posts

Re: End of an era for me: no more self-hosted git

#71

Post by embedding-shape »

M95D wrote: post #56
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I stopped trying to understand. Encountering a 404 on my site leads directly to a 1 year ban.

Damn, as someone who sometimes navigate by guessing URLs and rewriting them manually in the address bar, I hope more don't start doing this, I probably see at least one self-inflicted 404 per day at least.

Re: End of an era for me: no more self-hosted git

#72

Post by teeklp »

kimos wrote: post #64
tigerlily wrote: post #63

Earlier quoted context omitted.

How can I detect if my router is backdoored, or being used as a residential proxy?

If it’s legit you can ask your ISP if they sell use of your hardware. Or just don’t use the provided hardware and instead BYO router or modem or media converter or whatever. But I think what OP is implying is insecure hardware being infected by malware and access to that hardware sold as a service to disreputable actors. For that buy a good quality router and keep it up to date.

So you don't know? Why respond?

Re: End of an era for me: no more self-hosted git

#73

Post by mzajc »

tigerlily wrote: post #63
esseph wrote: post #50

Earlier quoted context omitted.

The dirty secret is a lot of them come through "residential proxies", aka backdoored home routers, iot devices with shitty security, etc. Basically the scrapers who are often also third party, go to these "companies" and buy access to these "residential proxies". Some are more... considerate than others. Why? Data. Every bit of it is it might be valuable. And not to sound tin foil hatty, but we are getting closer to…

How can I detect if my router is backdoored, or being used as a residential proxy?

I'm dealing with such attack, so if you'd like, you can send me IPv4 addresses, and I'll grep my logs for them. Email address is on the website linked on my profile.

As for what you can do on your own, it really depends on your network. OpenWRT routers can run tcpdump, so you can check for suspicious connections or DNS requests, but it gets really hard to tell if you have lots of cloud-tethered devices at home. IoT, browser extensions, and smartphone applications are the usual suspects.

Re: End of an era for me: no more self-hosted git

#74

Post by rozab »

embedding-shape wrote: post #70
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I just threw up a public Forjego instance for some lightweight collaboration. About 2 minutes after the certificate was created, I'm guessing they picked up the instance from the transparency logs for certificates, and started going through every commit and so on from the two repositories I had added. Watched it for a while, thinking eventually it'd end. It didn't, seemed like Claudebot and GPTBot (which was the only…

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPTbot, ClaudeBot, PerplexityBot, and others.

I added a robots.txt with explicit UAs for known scrapers (they seem to ignore wildcards), and after a few days the traffic died down completely and I've had no problem since.

Git frontends are basically a tarpit so are uniquely vulnerable to this, but I wonder if these folks actually tried a good robots.txt? I know it's wrong that they ignore wildcards, but it does seem to solve the issue

Re: End of an era for me: no more self-hosted git

#76

Post by bob1029 »

rozab wrote: post #74
embedding-shape wrote: post #70

Earlier quoted context omitted.

I just threw up a public Forjego instance for some lightweight collaboration. About 2 minutes after the certificate was created, I'm guessing they picked up the instance from the transparency logs for certificates, and started going through every commit and so on from the two repositories I had added. Watched it for a while, thinking eventually it'd end. It didn't, seemed like Claudebot and GPTBot (which was the only…

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPTbot, ClaudeBot, PerplexityBot, and others. I added a robots.txt with explicit UAs for known scrapers (they seem to ignore wildcards), and after a few days the traffic died down completely and I've had no problem since. Git frontends are basically a tarpit so a…

> I wonder if these folks actually tried a good robots.txt?

I suspect that some of these folks are not interested in a proper solution. Being able to vaguely claim that the AI boogeyman is oppressing us has turned into quite the pastime.

Re: End of an era for me: no more self-hosted git

#77

Post by t312227 »

hello,

as always: imho. (!)

idk ... i just put a http basic-auth in front of my gitweb instance years ago.

if i really ever want to put git-repositories into the open web again i either push them to some portal - github, gitlab, ... - or start thinking about how to solve this ;))

just my 0.02€

Re: End of an era for me: no more self-hosted git

#78

Post by Imustaskforhelp »

embedding-shape wrote: post #70
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I just threw up a public Forjego instance for some lightweight collaboration. About 2 minutes after the certificate was created, I'm guessing they picked up the instance from the transparency logs for certificates, and started going through every commit and so on from the two repositories I had added. Watched it for a while, thinking eventually it'd end. It didn't, seemed like Claudebot and GPTBot (which was the only…

Huh, I had a gitea instance in the public web on one of my netcup vps's. I didn't set any logs and was using cloudflare tunnels (with a custom bash script which makes cf tunnels expose PORT SUBDOMAIN).

Maybe its time for me to go ahead and start it again with logs to see if there are any logs.

I will maybe test it in all three 1) With CF tunnels + AI Block, 2) Only CF tunnels, 3) On a static IP directly. Maybe you can try the experiment too and we can compare our findings (also saying because I am lazy and I had misconfigured that cf tunnel so when it quit, I was too lazy to restart the vps given I just use it as a playground and just wanted to play around self hosting but maybe I will do it again now)

Re: End of an era for me: no more self-hosted git

#79

Post by snorremd »

I've recently been setting up web servers like Forgejo and Mattermost to service my own and friends' needs. I ended up setting up Crowdsec to parse and analyse access logs from Traefik to block bad actors that way. So when someone produces a bunch of 4XX codes in a short timeframe I assume that IP is malicious and can be banned for a couple of hours. Seems to deter a lot of random scraping. Doesn't stop well behaved crawlers though which should only produce 200-codes.

I'm actually not sure how I would go about stopping AI crawlers that are reasonably well behaved considering they apparently don't identify themselves correctly and will ignore robots.txt.

Re: End of an era for me: no more self-hosted git

#80

Post by madduci »

t312227 wrote: post #77

hello, as always: imho. (!) idk ... i just put a http basic-auth in front of my gitweb instance years ago. if i really ever want to put git-repositories into the open web again i either push them to some portal - github, gitlab, ... - or start thinking about how to solve this ;)) just my 0.02€

I've put everything behind a Wireguard Server, so if I need something, I can access to it through VPN and AI can't do anything
Post reply on HN