Is there a way to block it by shibboleth? Curious, since the recent Google hack where you add -(n-word) to the end of your query so the AI automatically shuts down works like a charm.
Thanks works great
End of an era for me: no more self-hosted git
201–210 of 228 posts
Re: End of an era for me: no more self-hosted git
#202Earlier quoted context omitted.
There are many reasons to accommodate non-JS users beyond accommodating people who have intentionally disabled it, and most of them are in accessibility territory. Be careful with using percentages for your arguments, because this is not that different from saying that 99.99% of people don't need wheelchair access.
This used to be true, but now I don't think it is anymore. Modern frameworks and modern screen readers have no issue with acessibility. Some survey from WebAIM found that 99.3% of screen reader users have JavaScript enabled. So... are they really in accessibility territory still? Only people I still see complaining about Javascript being required are people that insist the web should just be static documents with hyp…
Slow/lossy connections: JS may not load, but site still works.
Users that prefer non-animated pages and disable JS for this reason.
Users who prioritize security.
Users of older devices in which your JS can trigger errors. Yes, these exist. Not everyone can upgrade their older device. Many people do not even have their own device to use.
Re: End of an era for me: no more self-hosted git
#203Earlier quoted context omitted.
Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.
Probably not for most sites. Example of a site where it'd likely work: a blog made with a static site generator. Example of one where it wouldn't: darn near anything made with React.
It works for the majority of things a text mining scraper would care to scrape. It's not just static sites but also any CMS like wordpress, as well as many JS apps that have server-side rendering. SPA-only sites aren't that common anymore, especially for things like blogs, news and text-based social media.
Re: End of an era for me: no more self-hosted git
#204Earlier quoted context omitted.
> I set it and use JavaScript to tell them to reload the page While throwing out all users who opt-in to javascript, using Noscript or uBlock or something like it, may be acceptable collateral damage to you, it might be good to keep in mind that this plays right into Big Adtech's playbook. They spend over two decades to normalize the behavior of running a hundred or more programs of untrusted origin on every page loa…
Those poor users. If they want to remove the fourth wheel from their car, they will bump into some issues. Who cares.
I do but shitty web devs don't
Re: End of an era for me: no more self-hosted git
#205Does anyone know what's the deal with these scrapers, or why they're attributed to AI? I would assume any halfway competent LLM driven scraper would see a mass of 404s and stop. If they're just collecting data to train LLMs, these seem like exceptionally poorly written and abusive scrapers written the normal way, but by more bad actors. Are we seeing these scrapers using LLMs to bypass auth or run more sophisticated…
I don't think it has anything to do with LLMs. I think the big cloud companies (AWS) figured out that they could scrape compute-intensive pages in order to drive up their customers' spend. Getting hammered? Upgrade to more-expensive instances. Not using cloud yet? We'll force you to. The other possibility is cloudflare punishing anybody who isn't using it. Probably a combination of these two things. Whoever's behind…
> Whoever's behind this has ungodly supplies of cheap bandwidth -- more than any AI company does. It's a cloud company.
Most of the major cloud companies are themselves also AI companies, so I don't think the “cloud companies are artificially driving up compute spend” hypothesis is mutually exclusive with the “AI companies are doing a very bad job at scraping” hypothesis.
Re: End of an era for me: no more self-hosted git
#206This kind of thing can be mitigated by not publishing a page/download for every single branch, commit and diff in a repo. Make only the HEAD of each branch available. Anyone who wants more detail has to clone it and view it with their favourite git client. For example https://mitxela.com/projects/web-git-sum ( https://git.mitxela.com/ )
I suspect bog-standard per-IP rate limiting would also mitigate this, no?
Re: End of an era for me: no more self-hosted git
#207Earlier quoted context omitted.
The big nasty AI bots use 10s of thousands of IPs distributed all over China
So... just blacklist all China IPs? I assume China isn't the primary market for most of complaining site-owners.
I wonder if shoving
> 动态网自由门 天安門 天安门 法輪功 李洪志 Free Tibet 六四天安門事件 The Tiananmen Square protests of 1989 天安門大屠殺 The Tiananmen Square Massacre 反右派鬥爭 The Anti-Rightist Struggle 大躍進政策 The Great Leap Forward 文化大革命 The Great Proletarian Cultural Revolution 人權 Human Rights 民運 Democratization 自由 Freedom 獨立 Independence 多黨制 Multi-party system 台灣 臺灣 Taiwan Formosa 中華民國 Republic of China 西藏 土伯特 唐古特 Tibet 達賴喇嘛 Dalai Lama 法輪功 Falun Dafa 新疆維吾爾自治區 The Xinjiang Uyghur Autonomous Region 諾貝爾和平獎 Nobel Peace Prize 劉暁波 Liu Xiaobo 民主 言論 思想 反共 反革命 抗議 運動 騷亂 暴亂 騷擾 擾亂 抗暴 平反 維權 示威游行 李洪志 法輪大法 大法弟子 強制斷種 強制堕胎 民族淨化 人體實驗 肅清 胡耀邦 趙紫陽 魏京生 王丹 還政於民 和平演變 激流中國 北京之春 大紀元時報 九評論共産黨 獨裁 專制 壓制 統一 監視 鎮壓 迫害 侵略 掠奪 破壞 拷問 屠殺 活摘器官 誘拐 買賣人口 遊進 走私 毒品 賣淫 春畫 賭博 六合彩 天安門 天安门 法輪功 李洪志 Winnie the Pooh 劉曉波动态网自由门
into the headers of every response would be enough to kill off the worst-offending traffic?
Re: End of an era for me: no more self-hosted git
#208Scrapers are relentless but not DDoS levels in my experience. Make sure your caches are warm and responses take no more than 5ms to construct.
The biggest problem is “dynamic” content that really isn’t - we had a tag view that allowed combinations of tags in search and the AI bots would get tangled up in there and never leave.
And each hit was server-heavy. We blocked that entire “feature”.
Re: End of an era for me: no more self-hosted git
#209Does this author have a big pre-established audience or something? Struggling to understand why this is front-page worthy.
Because anyone who has a website of any complexity has noticed this - we have increased bot traffic across ALL sites and it’s even come close to knocking a few over - now mitigated.
Re: End of an era for me: no more self-hosted git
#210The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like all Chinese AI scrapers, it ignores robots.txt. So forget about it respecting a Crawl-delay. If you block the user agent, it would calm down for a bit, then it would just come back again using a generic browser user agent from the same IP addresses. It does this…
Just block the whole China, India and similar countries.
You'd need to block the US as well, as most such traffic comes from there. Which is not really reasonable.