691–698 of 698 posts
Earlier quoted context omitted.
How so?
Why else would a thing be programmed to drop cookies after every seconds request?
> proxy SDK monetization Wait what? I never heard of that. I call that a botnet
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
Earlier quoted context omitted.
You mean shoving it onto the investors?
No they mean the people doing the actual work, I think.
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
Earlier quoted context omitted.
Why else would a thing be programmed to drop cookies after every seconds request?
Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got comp…