Earlier quoted context omitted.
I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table
>> but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. > I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table Big Company Crawlers using questionably sourced proxies - this seems striking. What can you share about it?
They worked on malware detection. The most likely reason is very obvious: if you only allow traffic from residential addresses to your Command & Control server, you make anti-malware research (which is most likely coming from either a datacenter or an office building) an awful lot harder - especially when you give non-residential IPs a different and harmless response instead of straight-up blocking them.