691–700 of 708 posts
Earlier quoted context omitted.
How so?
Why else would a thing be programmed to drop cookies after every seconds request?
> proxy SDK monetization Wait what? I never heard of that. I call that a botnet
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
Earlier quoted context omitted.
You mean shoving it onto the investors?
No they mean the people doing the actual work, I think.
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
Earlier quoted context omitted.
Why else would a thing be programmed to drop cookies after every seconds request?
Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got comp…
Earlier quoted context omitted.
See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…
If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking. So you can definitely block by IP, by fingerprint, both or neither
why do you feel like discussing features was relevant?
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…
My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got comp…