Does this author have a big pre-established audience or something? Struggling to understand why this is front-page worthy.
[dead]
End of an era for me: no more self-hosted git
11–20 of 228 posts
Re: End of an era for me: no more self-hosted git
#12The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.
They don't. I'm using Cloudflare and 90%+ of the traffic I'm getting are still broken scrapers, a lot of them coming through residential proxies. I don't know what they block, but they're not very good at that. Or, to be more fair: I think the scrapers have gotten really good at what they do because there's real money to be made.
Re: End of an era for me: no more self-hosted git
#13Does this author have a big pre-established audience or something? Struggling to understand why this is front-page worthy.
Well the fact that this supposed nobody is overwhelmed by AI scrapers should speak a lot about the issue no?
Re: End of an era for me: no more self-hosted git
#14Does anyone know what's the deal with these scrapers, or why they're attributed to AI?
I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors.
Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated flows? I have not worked on bot detection the last few years, but it was very common for residential proxy based scrapers to hammer sites for years, so I'm wondering what's different.
Re: End of an era for me: no more self-hosted git
#15I presume people have logs that indicate the source for them to place blame on AI scrapers. Is anybody making these available for analysis so we can see exactly who is doing this?
Re: End of an era for me: no more self-hosted git
#16The author of this post could solve their problem with Cloudflare or any of its numerous competitors. Cloudflare will even do it for free.
I'm pretty sure scrapers aren't supposed to act as low key DOS attacks
Re: End of an era for me: no more self-hosted git
#17[flagged]
Re: End of an era for me: no more self-hosted git
#18Does this author have a big pre-established audience or something? Struggling to understand why this is front-page worthy.
because he's unable to self-host git anymore because AI bots are hammering it to submit PRs. self-hosting was originally a "right" we had upon gaining access to the internet in the 90s, it was the main point of the hyper text transfer protocol.
Also converting the blog from something dynamic to a static site generator. I made the same switch partly for ease of maintenance, but a side benefit is it's more resilient to this horrible modern era of scrapers far outnumbering legitimate traffic.
It's painful to have your site offline because a scraper has channeled itself 17,000 layers deep through tag links (which are set to nofollow, and ignored in robots.txt, but the scraper doesn't care). And it's especially annoying when that happens on a daily basis.
Not everyone wants to put their site behind Cloudflare.
Re: End of an era for me: no more self-hosted git
#19Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
There's value to be had in ripping the copyright off your stuff so someone else can pass it off as their stuff. LLMs have no technical improvements so all they can do is throw more and more stolen data into it and hope it, somehow, crosses a nebulous "threshold" where it suddenly becomes actually profitable to use and sell.
It's a race to the bottom. What's different is we're much closer to the bottom now.
Re: End of an era for me: no more self-hosted git
#20Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
I’m guessing, but I think a big portion of AI requests now come from agents pulling data specifically to answer a user’s question. I don’t think that data is collected mainly for training now but are mostly retrieved and fed into LLMs so they can generate the response. Thus so many repeated requests.