511–520 of 694 posts
Earlier quoted context omitted.
Everyone ignores robots.txt because every robots.txt blocks everyone.
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
Earlier quoted context omitted.
No point, they are all unique requests.
I think he means, get cloudflare to cache your content, so the traffic never reaches your servers to begin with. Assuming your sites content is cacheable by cloudflare. I agree it's a sad state of affairs if you have to rely on a 3rd party.. Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it ha…
Earlier quoted context omitted.
why is this not the answer? then you can also rate limit each cookie as well.
it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...
Earlier quoted context omitted.
You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?
> just > [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions] Never going to happen.
Earlier quoted context omitted.
It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely? > Us PHP developers I can tell.
it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.
Earlier quoted context omitted.
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust. Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
And yet you have not addressed the point that many report that tricks like Anubis work. If they are so stupid an idea that they could never work, why do they seem to having the desired effect?
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.
Earlier quoted context omitted.
Or poll a merkle tree root hash, like, um, git
What I'm after is a way to get away from polling by server name. With a sufficiently interested audience I ought to be able to be offline except when I need to update the merkle root, meanwhile users can just serve the content to each other.
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
- what else?
[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...
[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...
Earlier quoted context omitted.
And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare. I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I…
Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
Earlier quoted context omitted.
It does. It's literally a felony but for some reason not a single person has pressed charges.
It's because the companies doing this bought the current government.