71–80 of 291 posts
I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...
> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
I'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
You mean like this? [2025-03-19] https://blog.cloudflare.com/ai-labyrinth/ > Trapping misbehaving bots in an AI Labyrinth > Today, we’re excited to announce AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives.
> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
Companies running those bots have more than enough resources
For highly valuable information, they might throw the GDP of a small country at scraping your site. But most information isn't worth that.
And there are a lot of bad actors who don't have the resources you're thinking of that are trying to compete with the big guys on a budget. This would cut them out of the equation.
> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.
Something similar to proof-of-work but on a much smaller scale than Bitcoin.
Earlier quoted context omitted.
This has me wondering what it would take to do a bcrypt style slow hashing requirement to retrieve data from a site. Something fast enough that a single mobile client for a user wouldn't really feel the difference. But an automated scraper would get bogged down in the calculations. Data is presented to the user with multiple layers of encryption that they use their personal key to decrypt. This might add an extra 200…
The problem is that the server also has to do the work. Fine for an infrequent auth challenge. Not so fine for every single data request.
Excuse my ignorance, but is it time to update the open source licenses in the light of this behavior? If so, what should the evolved license wording be? I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...
A license can help as well, but what's a license without enforcement? These companies are simply treating the courts as a cost to do business.