Avoiding bot detection: How to scrape the web without getting blocked?

The majority of them want to do something completely benign like see a BBC show in the US, or watch an American football show in the UK, and the one defining feature of capitalism is its many contradictory faces.

One corporation wants to arbitrarily limit its customer base, and the other corporation wants to arbitrarily limit what data its customer base can see. In between the two is a space big enough to drive a Mack truck or a lorry through, depending on where you're from...

One might justify this until the one corporation merges with the other and then you have a situation where the same corporation wants to do two different things to the same pool of users.

There is vast amount of profit available in doing just that (see e.g. GOOG and FB market cap). Even companies that truly have no intention of exploiting data collected as a side-effect of whatever product line they do, nearly always eventually end up going for that profit line. Because passing up on more income merely due to moral considerations is too much of a temptation for a company to be able to resist in the long term.

I generally appreciate registered API integrations, but the trouble is, for the most sites that are most problematic for benign automation, they usually don't have enough demand or revenue to justify well-maintained APIs.

I tend to think the solution is more to somehow make the market prefer more decentralized solutions, preferably federated. Not having one big target for bad actors means much less effort applied to attacking any one of the targets.

Dating might be a bit off topic, but I can see both sides as well. Women have genuine risks, and are very justified in taking precautions. But for the majority of decent guys, it can be tiresome to be constantly treated like you're an evil violent stalker. Maybe it needs a similar solution - a movement to local connections where people can have reputations that you can trust.

So you do some thing which once a day scrapes a site and pulls off some data that you use in your thing. Maybe you talk about it to friends, or you have this thing as one of your github repositories. Some of your friends download the repository and also start using your thing. They talk to people about how cool your thing is, or what it does and the nice convenience of automating something that you used to have to do manually.

There are 86,400 seconds in a 24 hour period, probably folks won't change your code at all at first, and as it diffuses into the community some webmaster starts seeing this weird spike of queries that happen once a day at some time. Different addresses but always the same kind of request.

It's not a problem when its like 10 or 20 qps burst but when it starts getting up to 100 - 200 or worse 1000 - 2000, it causes the system to perhaps spin up additional instances that it isn't going to need after the burst and waste money. So the webmaster starts denying those requests with a 404.

Now sometimes your code works and sometimes it doesn't but you don't know that the webmaster is fighting you yet. Maybe eventually you start randomly varying the request time, or the people who have copied your thing are in more varied time zones so you it starts getting spread over the day.

now the webmaster is seeing bursts of traffic nearly every hour on the hour and that is weird so a more aggressive mitigation strategy is enacted.

People using your thing complain that it keeps breaking so you look into it and realize that the site is trying to block your requests. Perhaps you don't understand why this is, or perhaps you do and don't care, either way you come up with some strategies that avoid the block (maybe your rotate the user-agent or something).

Now the query traffic is spiking again and the webmaster is getting complaints that this 'bot traffic' is resulting in useless AWS fees because it isn't part of the revenue traffic and it is forcing the service to add more resources for their customers.

Not all scrapers are malicious, but my experience is that it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching) and because its all open source it spreads around.