Web Scraping Is Vital to Democracy

There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do.

Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them access becomes legally binding.

In the offline world, we derive great benefit from following the second model. Indeed, if only measures that successfully prevent people from entering your house without permissions were to count, you wouldn't need laws in the first place! You would, however, need a bunker. Which is quite a bit more expansive than a functioning legal system.

In that offline world, we have created all sorts of additional rules to balance rights for specific situations, and we rely on a canon of expectations that say, for example, that it's usually not ok to enter a private house, but you don't need explicit permission to enter a supermarket.

These are still developing for the online world, and your idea of the "ecosystem" hints at that. But you're just taking from those contradictory ideas above to arrive at the outcome you intuitively feel is "just": a bit of might-is-right when it comes to "whatever is online is fair game", followed by principled ideas of rights and obligations when websites try to defend themselves in that jungle of yours.

What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.

Other than that, you're mixing civil and criminal law rather liberally. I agree that it would be insane to create criminal liability for run-of-the-mill violations of ToS, and there is a decision from the MySpace era saying as much (and not even involving any changes to those ToS).

But once you have been specifically asked to not do something, by any means that would reasonably get that message across (so, not just C&D), it becomes... murky?

There is no middle ground _in the protocol_ and that's why I explicitly said hacks. Any system can be rehashed into anything else with unlimited extra layers on top of it - by your definition everything is everything.

> What's really needed is something that can, for example, distinguish between a journalist scraping Facebook to map out a terror network vs. some other entity scraping Facebook to sell your embarrassing photos to the highest bidder ten years down the road.

Sorry but that sounds uneforcable and rather absurd. We have the framework in place already - if you don't want something to be public don't put it out in the public _explicitly_. To add we already have legal framework in place for copyrighted and/or protected content like photos and against any sort of malicious attacks like ddos.

Our web is getting extremely centralized and most of these majors are natural monopolies: google search becomes stronger the more data it has - google can scrape the entire web freely yet it's competition can't; facebook becomes stronger the more data is has etc. etc. One way to restore balance is to ensure that public data remains public and the ecosystem can have healthy competition and growth otherwise we're moving to a very dystopian corporate owned world.

Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it.

As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from people. There really is no one-fits-all solution here, because anything that allows the user to intercept and monitor SSL communication can automatically be used by an attacker! :(