Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

151–160 of 228 posts

Re: End of an era for me: no more self-hosted git

#151

Post by shakna »

kstrauser wrote: post #126
xorcist wrote: post #120

Earlier quoted context omitted.

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…

Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of us…

> JS is as fundamental to current design as CSS.

I think this hits the crux of the trend fairly well.

And is why I have so many workarounds to shitty JS in my user files.

Because I can't see your CSS, either.

Re: End of an era for me: no more self-hosted git

#152

Post by groby_b »

everybodyknows wrote: post #149
Tharre wrote: post #91

Earlier quoted context omitted.

> Does anyone know what's the deal with these scrapers, or why they're attributed to AI? You don't really need to guess, it's obvious from the access logs. I realize not everyone runs their own server, so here are a couple excerpts from mine to illustrate: - "meta-externalagent/1.1 + https://developers.facebook.com/docs/sharing/webmasters/craw... )" - "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cl…

Following your link above, https://openai.com/gptbot > ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. So, not AI training in this case, nor any other large-batch scraping, but rather inference-time Retrieval Augmented Generation, with the "retrieval" happening over the web?

Likely, at least for some. I've caught various chatbots/CLI harnesses more than once inspecting a github repo file by file (often multiple times, because context rot)

But the sheer volume makes it unlikely that's the only reason. It's not like everybody has constantly questions bout the same tiny website.

Re: End of an era for me: no more self-hosted git

#153

Post by bandie91 »

Imustaskforhelp wrote: post #88
moebrowne wrote: post #83

This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )

I got another interesting idea from this and another comment but what if we combine this with ssh git clients/websites with the normal ability. maybe something like https://ssheasy.com/ or similar could also be used? or maybe even a gotty/xterm instance which could automatically ssh/get a tui like interface. I feel as if this would for all scrapers be enough?

i'm working on something similar: instead of web-based ssh client, it's a web-based git client UI - you can "checkout" repos, browse commits, tree, read individual files, etc. with no server-side code at all; git objects are fetched and parsed on client-side. first target is the dumb-http git protocol, so people can host git repos on static websites, and visitors don´t need to clone by a local git client to peek in.

https://bandie91.github.io/dumb-http-git-browser-js-app/ui.h...

Re: End of an era for me: no more self-hosted git

#154

Post by hxtk »

tuhgdetzhh wrote: post #137
QuiDortDine wrote: post #108

Earlier quoted context omitted.

I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.

I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.

Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.

Re: End of an era for me: no more self-hosted git

#155

Post by skrtskrt »

stefanka wrote: post #116
rozab wrote: post #74

Earlier quoted context omitted.

I had the same issue when I first put up my gitea instance. The bots found the domain through cert registration in minutes, before there were any backlinks. GPTbot, ClaudeBot, PerplexityBot, and others. I added a robots.txt with explicit UAs for known scrapers (they seem to ignore wildcards), and after a few days the traffic died down completely and I've had no problem since. Git frontends are basically a tarpit so a…

Where does one find a good robots.txt? Are there any well maintained out there?

Cloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.

Re: End of an era for me: no more self-hosted git

#156

Post by the_biot »

esseph wrote: post #106
the_biot wrote: post #103

Earlier quoted context omitted.

Has this actually been investigated and proven to be true? I see allegations, but no facts really. It seems to me to be just as likely that people are installing LLM chatbot apps that do the occasional bit of scraping work on the sly, covered by some agreed EULA.

Seriously, go to Google. Search for: "residential proxy" ai data scraping. Start reading through thousands of articles.

That's the worst thing I've seen all week. The DDoS networks of 20 years ago, now out in the open and presented as real business.

Thanks for the info, wish I didn't know :-(

Re: End of an era for me: no more self-hosted git

#157

Post by qwery »

kstrauser wrote: post #126
xorcist wrote: post #120

Earlier quoted context omitted.

> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…

Oh please. That ship has sailed. I'm marginally sympathetic to people who don't run JavaScript on their browsers for a variety of reasons, but they've deliberately opted out of the de facto modern web. JS is as fundamental to current design as CSS. If you turn it off, things might work, but almost no one is testing that setup, nor should they reasonably be expected to. This has zero to do with Adtech for 99.99% of us…

the recent google report claimed that less than 0.1% of users have javascript disabled ... like for every website, or just some, or?

your PNG/GIF thing is nonsense (false equivalence, at least) and seems like deliberate attempt to insult

> I'm marginally sympathetic

you say that as if they've done some harm to you or anyone else. outside of these three words, you actually seem to see anyone doing this as completely invalid and that the correct course of action is to act like they don't exist.

Re: End of an era for me: no more self-hosted git

#158

Post by mqus »

I "solved" this by adding a fail2ban rule for everyone accessing specific commits (no one does that 3 times in a row) and then blocking the following ASs completely (just too many IPs coming from those, feel free to look them up yourself): 136907 23724 9808 4808 37963 45102. And after that: sweet silence.

How to block ASs? Just write a small script that queries all of their subnets once (even if it changes, its not so much to have an impact) and add them to a nft set (nft will take care of aggregating these into continouus blocks). Then just make nft reject requests from this set.

Re: End of an era for me: no more self-hosted git

#159

Post by HumanOstrich »

Imustaskforhelp wrote: post #88
moebrowne wrote: post #83

This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )

I got another interesting idea from this and another comment but what if we combine this with ssh git clients/websites with the normal ability. maybe something like https://ssheasy.com/ or similar could also be used? or maybe even a gotty/xterm instance which could automatically ssh/get a tui like interface. I feel as if this would for all scrapers be enough?

[flagged]

Re: End of an era for me: no more self-hosted git

#160

Post by zerocrates »

simonw wrote: post #25
data-ottawa wrote: post #14

Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…

I would love to understand this. Just a few years ago badly behaved scrapers were rare enough not to be worth worrying about. Today they are such a menace that hooking any dynamic site up to a pay-to-scale hosting platform like Vercel or Cloud Run can trigger terrifying bills on very short notice. "It's for AI" feels like lazy reasoning for me... but what IS it for? One guess: maybe there's enough of a market now for…

I would say there's a couple aspects.

The crawlers for the big famous names in AI are all less well behaved and more voracious than say, Googlebot. Though this is all somewhat muddied by companies that ran the former "good" crawlers all also being in the AI business and sometimes trying to piggyback on people having allowed or whitelisted their search crawling User-Agent, mostly this has settled a little where they're separating Googlebot from GoogleOther, facebookexternalhit from meta-externalagent, etc. This was an earlier "wave" of increased crawling that was obviously attributable to AI development. In some cases it's still problematic but this is generally more manageable.

The other stuff, the ones that are using every User-Agent under the sun and a zillion datacenter IPs and residential IPs and rotate their requests constantly so all your naive and formerly-ok rate-based blocking is useless... that stuff is definitely being tagged as "for AI" on the basis of circumstantial evidence. But from the timing of when it seemed to start, the amount of traffic and addresses, I don't have any problem guessing with pretty high confidence that this is AI. To your question of "who are the customers"... who's got all the money in the world sloshing around at their fingertips and could use a whole bunch of scraped pages about ~everything? Call it lazy reasoning if you'd like.

How much this traces back ultimately to the big familiar brand names vs. would-be upstarts, I don't know. But a lot of sites are blocking their crawlers that admit who they are, so would I be surprised to see that they're also paying some shady subcontractors for scrapes and don't particularly care about the methods? Not really.

Post reply on HN