What you've failed to mention is the criteria used to determine if a usage is indeed "fair". There are 4 basic criteria[0] but can be summarized as "If the usage doesn't affect the market for the original work, is substantially transformative, is proportionally insignificant or is used for critique/parody then it is fair". Or, at the risk of over simplifying it: "Does the usage grant a net public benefit without significantly hurting the copyright holders ability to make money?".
>Can I send an email to Netflix and tell them "Hey, if you don't want me to copy your shows, please add this in your page's HEAD element: "?
Actually, under fair use you certainly can make a personal copy (see Betamax case). If you distribute the work you would likely run afoul of the criteria summarized above.
The robots.txt relevancy is being over stated in your argument. The main criteria used in this case is summarized above. The fact that Google provides an opt-out mechanism is a secondary, supporting argument.
>What if I started indexing and rehosting thumbnails? I can assure you that I would get C&D'd almost immediately
A determination of infringement would depend entirely on the context as related to the afore mentioned criteria. The fact that someone might try to sue is a product of the terrible system in general and you're absolutely right - as with any legal matter the entity with the deeper pockets can often bully the other guy into submission.
>In Craigslist v. 3Taps, while primarily a CFAA case, 3Taps was found to be infringing copyrights
My understanding is that the copyright part of the case was thrown out [1] and thus was settled solely around CFAA matters.
>In Ticketmaster v. RMG Technologies , RMG was found to infringe just by parsing a page.
I agree that the logic used for the judgement is absurd (for reasons that are plainly obvious to any HN user). But it's less clear whether the case would meet fair use criteria outlined above should it have come to that. My guess is that it wouldn't qualify since the usage affects the copyright holders ability to make money on the work and doesn't meet any of the other criteria for Fair Use.
>Facebook v. Power Ventures
This is not a case involving a defense of fair use (as far as I can tell). Facebook even acknowledged the users owned the data and had a right to it. The defendant was actually found to be violating CFAA and CAN-SPAM acts.
>It seems Google is the only entity capable of making unauthorized copies and then getting courts to agree that it's fair use. For the rest of us, it's infringement
Provably false [2]. It sounds like perhaps your personal experience has soured your opinion on the matter? That's understandable. But none of the evidence you've cited supports the argument that Google is infringing copyrights in its core activities nor that Google is the only entity where copyright laws and fair use legislation don't apply.
PS: To be clear, my argument revolves specifically around copyright infringement and fair use. I don't have enough understanding of other, separate legislation like CFAA to comment on that except to say that it seems overly broad and unrealistic. But that's another topic. I'm specifically arguing against calling Google a copyright infringer in a broad sense which is what you've done. That's not been proven.
[0] https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors
[1] https://techcrunch.com/2013/04/30/craigslist-3taps-lawsuit-d...
[2] http://fairuse.stanford.edu/overview/fair-use/cases/