21–30 of 52 posts
What year are they reminiscing about here, 1999? Nothing has respected robots.txt in over twenty years.
Nofollow isn't an anti-bot measure; it's supposed to inform search engines that you don't vouch for the linked content (don't wish to boost its rank). Nofollow doesn't mean "you must not follow this if you're a crawler".
Earlier quoted context omitted.
AI bots operators are financially incentivized to not be good citizens, they want as much data as possible as fast as possible and don't care who they piss off in the process. Plus for now at least they have effectively unlimited money to throw at bandwidth, storage, IP addresses, crawling with full-blown headless browsers, etc.
And it gets worse. For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents). But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying so…
I have temporarily disabled my HTTP server for now. (I set up port knocking for a day, but I got rid of it due to a kernel panic.) My issue is not to prevent anyone from obtaining a copy if they want to do, and I want to ensure that users can use curl, Lynx, and other programs; I do not want to require JavaScripts, CSS, Firefox, Google, etc. My problem is that these LLM scraping bots are badly behaved, making many re…
How can you tell they are LLM bots?
The only way--the only way--to solve these issues is with web servers requiring that all clients authenticate with a credential that is provably tied to a real-world entity--person or corporate entity--so that legal recourse is available to the server owner when abuse occurs. The internet is no longer high-trust; we're running web servers the same way we'd run an honor-system store where people just come in and steal…
Earlier quoted context omitted.
And it gets worse. For now they are probably paying to use residential IP addresses that they get from other services that sell them (and these services get them from people who willingly sell some of their bandwidth for cents). But I think it won't be long before we start seeing the AI companies having each their own swarm of residential IP addresses by selling themselves a browser extension or mobile app, saying so…
OpenAI has ‘already’ got a browser extension. Who knows when this is ‘enabled’. We already had the ‘honey’ debacle with Amazon/ebay referral link stealing
You could test this theory pretty easily by monitoring traffic... (I haven't, but maybe someone has?)
Can we at least get rid of CAPTCHAs now since they clearly don't work?
I can't really understand the outrage here, this problem of scraping to the point of being DDOSed, which is what the author seems to contend, has existed since forever.
Uhmm.... Just use harder Catpchas? I can't really understand the outrage here, this problem of scraping to the point of being DDOSed, which is what the author seems to contend, has existed since forever.
> They are using commercial services such as Cloudflare to outsource their bot-blocking and captchas, without knowing for sure what these services are blocking, how they're doing it, or whether user privacy and accessibility is being flushed down the toilet. But nothing seems to offer anything but temporary relief.
Also:
> The current generation of bots is mindless. They use as many connections as you have room for. If you add capacity, they just ramp up their requests. They use randomly generated user-agent strings. They come from large blocks of IP addresses. They get trapped in endless hallways. I observed one bot asking for 200,000 nofollow redirect links pointing at Onedrive, Google Drive and Dropbox. (which of course didn't work, but Onedrive decided to stop serving our Canadian human users). They use up server resources - one speaker at Code4lib described a bug where software they were running was using 32 bit integers for session identifiers, and it ran out!
This could be at least partially solved by starting legal and cybersec (bulk blocks, flagging SDKs as malware) action against botnets for rent[0], forcing their SDKs out of app stores[1].
0 – https://spur.us/residential-proxies-the-legal-botnet-that-no... 1 – https://datadome.co/bot-management-protection/how-proxy-prov...