Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

181–190 of 228 posts

Re: End of an era for me: no more self-hosted git

#181

Post by raw_anon_1111 »

xorcist wrote: post #120
kstrauser wrote: post #93

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…

You mean all 3?

I could care less about serving users who don’t want to enable JS in 2026. They aren’t worth my development times

Re: End of an era for me: no more self-hosted git

#182

Post by sh34r »

dirkc wrote: post #121
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I'm hazarding a guess that there are many AI startups that focus on building datasets with the aim to sell those datasets. Still doesn't make total sense, since doing it badly would only hurt them, but maybe they don't really care about the product / outcome, they're just capturing their bit of the AI goldrush?

I guess the AI companies finally figured out they’re supposed to buy their stolen datasets from a shell company spun up by the most unsavory character within two degrees of the CEO. Every CEO has a drug dealer, and every CEO drug dealer knows the greasy grey hat dude running a data laundry “startup.” The VCs usually know some private equity dons who run the same racket to do bust out fraud, too.

It’s truly unbelievable that OpenAI and Anthropic were so sloppy. Pirating all that copyrighted media and not even bothering to hide behind one layer of indirection. Amateurs.

So yeah… it’s what, five years’ worth of pent up demand for organized crime, hitting the market everywhere all at once? I’m surprised the request volume isn’t higher!

Re: End of an era for me: no more self-hosted git

#183

Post by kstrauser »

raw_anon_1111 wrote: post #180
kstrauser wrote: post #139

Earlier quoted context omitted.

It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.

Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.

Probably not for most sites. Example of a site where it'd likely work: a blog made with a static site generator. Example of one where it wouldn't: darn near anything made with React.

Re: End of an era for me: no more self-hosted git

#184

Post by aspenmayer »

simonw wrote: post #26
Jaxkr wrote: post #2

The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.

Cloudflare won't save you from this - see my comment here: https://news.ycombinator.com/item?id=46969751#46970522

Parent of your comment became [flagged][dead], which broke your in-context link.

A direct link works, however:

https://news.ycombinator.com/item?id=46970522

Re: End of an era for me: no more self-hosted git

#185

Post by diath »

reactordev wrote: post #141
PaulDavisThe1st wrote: post #133

Earlier quoted context omitted.

Using gitea does not help if you goal is to allow non-auth'ed read-only access to the repo from a web browser. The scrapers use that to hit up every individual commit, over and over and over. We used nginx config to prevent access to individual commits, while still leaving the "rest" of what gitea makes available read-only for non-auth'ed access unaffected.

Yuk… http { # ... other http settings limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; # ... } server { # ... other server settings location / { limit_req zone=mylimit burst=20 nodelay; # ... proxy_pass or other location-specific settings } } Rate limit read-only access at the very least. I know this is a hard problem for open source projects that have relied on web access like this for a while. Anubis…

Easier said than done, I have 700k requests from bots in my access.log coming from 15k different IP addresses.

:: ~/website ‹master*› » rg '(GPTBot|ClaudeBot|Bytespider|Amazonbot)' access.log | awk '{print $1}' | sort -u | wc -l

15163

Re: End of an era for me: no more self-hosted git

#186

Post by t312227 »

t312227 wrote: post #77

hello, as always: imho. (!) idk ... i just put a http basic-auth in front of my gitweb instance years ago. if i really ever want to put git-repositories into the open web again i either push them to some portal - github, gitlab, ... - or start thinking about how to solve this ;)) just my 0.02€

hello,

as always: imho. (!)

btw. thanks for the downvote.

its for sure better to kill your own infrastructure because of some AI crawlers - buhuuuu ... bad bots!! - than to solve your problem with a stupid simple but effective solution.

just as an idea: if i had to host public repositories i would think about how to disable costly operations - searches etc. - for anonymous access ... like github did.

just my 0.02€

Re: End of an era for me: no more self-hosted git

#188

Post by 7bit »

xorcist wrote: post #120
kstrauser wrote: post #93

I cut traffic to my Forgejo server from about 600K request per day to about 1000: https://honeypot.net/2025/12/22/i-read-yann-espositos-blog.h... 1. Anubis is a miracle. 2. Because most scrapers suck, I require all requests to include a shibboleth cookie, and if they don’t, I set it and use JavaScript to tell them to reload the page. Real browsers don’t bat an eye at this. Most scrapers can’t manage it. (This wasn’t…

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…

Those poor users. If they want to remove the fourth wheel from their car, they will bump into some issues. Who cares.

Re: End of an era for me: no more self-hosted git

#189

Post by Imustaskforhelp »

HumanOstrich wrote: post #159
Imustaskforhelp wrote: post #88

Earlier quoted context omitted.

I got another interesting idea from this and another comment but what if we combine this with ssh git clients/websites with the normal ability. maybe something like https://ssheasy.com/ or similar could also be used? or maybe even a gotty/xterm instance which could automatically ssh/get a tui like interface. I feel as if this would for all scrapers be enough?

[flagged]

This is really not the type of legacy that I want to leave behind on hackernews but then again, I have been vocal that I just write what I think. Literally. It has its flaws but I am not sugar coating it.

Sometimes I am unable to explain myself but the thing is that I write on HN to point out of some idea, some discussion. It's better written here than lost and yes most of my ideas might be incoherent but they make perfect sense to me in the moment, its quite hard to explain.

It seems that you have made an opinion & judgement about me and that's okay. I don't wish to change it.

I suppose while writing your comment, you must have been quite pissed to write it. Sorry for pissing you off in such sense, That wasn't my intention but I do hope that you can realize that your comment comes across as rude and quite frankly, I don't know how to respond to it and I don't want to throw myself to this level or continue in an argumentative tone.

We are more common than different actually. I suppose we both love open source and might share many hobbies. The difference is small when you think about it.

I hope that instead of fighting on our differences, we can work with our agreements. Teach me instead of such tone for I am interested in learning & let's hope that both of us and everyone can make a better future for the world & everyone living in it :D

Have a nice day, my friend. Hope the future's good for ya!

So answer me this, what's your favourite open source project and why? and I will answer mine when you respond later :]

Re: End of an era for me: no more self-hosted git

#190

Post by longor1996 »

bcrl wrote: post #124
mzajc wrote: post #92

Earlier quoted context omitted.

Yes, the attack is continuous. The rate fluctuates a lot, even within a day. It's definitely an anomaly, because eg. from 2025-08-15 to 2025-10-05 I saw zero days with more than 10k requests. Here's a histogram of the past 2 weeks plus today. 2026-01-28 21'460 2026-01-29 27'770 2026-01-30 53'886 2026-01-31 100'114 # 2026-02-01 132'460 # 2026-02-02 73'933 2026-02-03 540'176 ##### 2026-02-04 999'464 ######### 2026-02-0…

It's plausible that the AI companies have given up storing data for training runs and just stream it off the Internet directly now. It's probably cheaper to stream than buying more SSDs and HDDs from a supply constrained supply chain at this point.

That this is a plausible explanation is... beyond horrifying to me.
Post reply on HN