41–50 of 55 posts
Have you considered serving a zip bomb to this user agent?
I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.
https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LS...
1. Put a note in robots.txt that says "By accessing this file more than one time per second you agree to pay a fee of $0.1 per access plus an additional $0.1 for each previous access each day. This fee will be charged on a per access basis." 2. Run a program that logs the number for Facebook requests and prints a summary and bill. 2. Then get a stamp, envelope and write out a bill for the first day, call it a demand…
Do you think there's a contract created by your robots.txt comment?
1. Put a note in robots.txt that says "By accessing this file more than one time per second you agree to pay a fee of $0.1 per access plus an additional $0.1 for each previous access each day. This fee will be charged on a per access basis." 2. Run a program that logs the number for Facebook requests and prints a summary and bill. 2. Then get a stamp, envelope and write out a bill for the first day, call it a demand…
1. Put a note in robots.txt that says "By accessing this file more than one time per second you agree to pay a fee of $0.1 per access plus an additional $0.1 for each previous access each day. This fee will be charged on a per access basis." 2. Run a program that logs the number for Facebook requests and prints a summary and bill. 2. Then get a stamp, envelope and write out a bill for the first day, call it a demand…
I recently started maintaining a MediaWiki instance for a niche hobbyist community and we'd been struggling with poor server performance. I didn't set the server up, so came into it assuming that the tiny amount of RAM the previous maintainer had given it was the problem. Turns out all of the major AI slop companies had been hounding our wiki constantly for months, and this had resulted in Apache spawning hundreds of…
1. Put a note in robots.txt that says "By accessing this file more than one time per second you agree to pay a fee of $0.1 per access plus an additional $0.1 for each previous access each day. This fee will be charged on a per access basis." 2. Run a program that logs the number for Facebook requests and prints a summary and bill. 2. Then get a stamp, envelope and write out a bill for the first day, call it a demand…
Earlier quoted context omitted.
> this had resulted in Apache spawning hundreds of instances, bringing the whole machine to a halt. Ugh, such a weird design. At least my experience has been you are better off setting Apache to always run the same number of instances, and tuning that number as appropriate rather than having the instance count fluctuate under load.
Yeah, I did suggest nginx in the past for this project and I think I'll consider it after this saga. For now I've set pretty low limits on Apache.
> Perhaps someone at their end screwed up a loop conditional, but you'd think some monitoring dashboard somewhere would have a warning pop up because of this. If you've been in any big company you'll know things perpetually run in a degraded, somewhat broken mode. They've even made up the term "error budget" because they can't be bothered to fix the broken shit so now there's an acceptable level of brokenness.
Keep in mind that "fixing things" is essentially a Sisyphean task - no matter how much you do there's always more you can do. Just like adding features. You have to have some kind of guideline on when enough is enough.