Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End of an era for me: no more self-hosted git

kraxel.org

Post reply on HN
Print view View original

41–50 of 228 posts

Re: End of an era for me: no more self-hosted git

#41

Post by Semaphor »

Jaxkr wrote: post #2

The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.

For logging, statistics etc. we have the Cloudflare bot protection on the standard paid level, ignore all IPs not from Europe (rough geolocation), and still have over twice the amount of bots that we had ~2 years ago.

Re: End of an era for me: no more self-hosted git

#42

Post by kevin_thibedeau »

JohnTHaller wrote: post #30

The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like all Chinese AI scrapers, it ignores robots.txt. So forget about it respecting a Crawl-delay. If you block the user agent, it would calm down for a bit, then it would just come back again using a generic browser user agent from the same IP addresses. It does this…

Start blocking /16s.

Re: End of an era for me: no more self-hosted git

#44

Post by krick »

So, what's up with these bots, why am I hearing about that so often lately? I mean, DDoS atacks aren't a new thing, and, honestly, this is pretty much the reason why Cloudflare even exists, but I'd expect OpenAI bots (or whatever this is now) to be a little bit easier to deal with, no? Like, simply having resonable aggressive fail2ban policy? Or do they really behave like a botnet, where each request comes from different IP from a different network? How? Why? What is this thing?

Re: End of an era for me: no more self-hosted git

#45

Post by krick »

JohnTHaller wrote: post #31
Lerc wrote: post #15

I presume people have logs that indicate the source for them to place blame on AI scrapers. Is anybody making these available for analysis so we can see exactly who is doing this?

The big nasty AI bots use 10s of thousands of IPs distributed all over China

So... just blacklist all China IPs? I assume China isn't the primary market for most of complaining site-owners.

Re: End of an era for me: no more self-hosted git

#46

Post by recursivecaveat »

krick wrote: post #44

So, what's up with these bots, why am I hearing about that so often lately? I mean, DDoS atacks aren't a new thing, and, honestly, this is pretty much the reason why Cloudflare even exists, but I'd expect OpenAI bots (or whatever this is now) to be a little bit easier to deal with, no? Like, simply having resonable aggressive fail2ban policy? Or do they really behave like a botnet, where each request comes from diffe…

I doubt it's OpenAI. Maaaybe somebody who sells to OpenAI, but probably not. I think they're big enough to do this mostly in-house and properly. Before AI only big players would want a scrape of the entire internet, they could write quality bots, cooperate, behave themselves, etc. Now every 3rd tier lab wants that data and a billion startups want to sell it, so it's a wild west of bad behavior and bad implementations. They do use residential IP sets as well.

Re: End of an era for me: no more self-hosted git

#47

Post by watermelon0 »

vachina wrote: post #43

Scrapers are relentless but not DDoS levels in my experience. Make sure your caches are warm and responses take no more than 5ms to construct.

Great, now we need caching for something that's seldom (relatively speaking) used by people.

Let's not forget that scrapers can be quite stupid. For example, if you have phpBB installed, which by defaults puts session ID as query parameter if cookies are disabled, many scrapers will scrape every URL numerous times, with a different session ID. Cache also doesn't help you here, since URLs are unique per visitor.

Re: End of an era for me: no more self-hosted git

#48

Post by _zagj »

simonw wrote: post #22
oceanplexian wrote: post #17

[flagged]

A varnish cache won't help you if you're running something like a code forge where every commit has its own page - often more than one page, there's the page for the commit and then the page for "history from this commit" and a page for every one of the files that existed in the repo at the time of that commit... Then a poorly written crawler shows up and requests 10,000s of pages that haven't been requested recently…

Well, it's heartening to know that AI is making your life at least somewhat less enjoyable.

Re: End of an era for me: no more self-hosted git

#49

Post by _zagj »

oceanplexian wrote: post #17

[flagged]

Your post is pure victim-blaming, as well as normalizing an exploitative state of affairs (being aggressively DDOSed by poorly-behaved scrapers run by Big Tech that only take and never give back, unlike pre-AI search engines, which previously, at least, would previously send you traffic) that was unheard of until just a few years ago.

Re: End of an era for me: no more self-hosted git

#50

Post by esseph »

krick wrote: post #44

So, what's up with these bots, why am I hearing about that so often lately? I mean, DDoS atacks aren't a new thing, and, honestly, this is pretty much the reason why Cloudflare even exists, but I'd expect OpenAI bots (or whatever this is now) to be a little bit easier to deal with, no? Like, simply having resonable aggressive fail2ban policy? Or do they really behave like a botnet, where each request comes from diffe…

The dirty secret is a lot of them come through "residential proxies", aka backdoored home routers, iot devices with shitty security, etc. Basically the scrapers who are often also third party, go to these "companies" and buy access to these "residential proxies". Some are more... considerate than others.

Why? Data. Every bit of it is it might be valuable. And not to sound tin foil hatty, but we are getting closer to a post-quantum time (if we aren't already ).

Post reply on HN