The fact that my local CPU is a different ISA from the server is barely a consideration after I've already had to account for the fact that the database is local instead of remote, that I'm using PCIe-attached storage instead of a network-mounted volume, that my local workload is wildly different from the deployment environment's... Ultimately, cross-development is much more meaningful as a software problem than it is as a hardware problem — but, given the popularity of macOS as a development environment, it seems clear that we're happy developing on something unix-y and deploying to another brand of unix-y something-or-other.
Linus is then ignoring the single biggest factor in this equation: performance-per-watt. ARM offers better performance/watt numbers than anything Intel can provide right now. Recent offerings on the server space suggest ARM is managing to scale up without sacrificing too much on that front, and Intel doesn't seem to be keeping up.
If you're operating a data centre, power supply is the single hardest limit to your capacity to scale up. Putting up another building at a pre-existing site is cheaper and easier than finding a way to pipe more power into it, so it doesn't even matter that much that servers are individually less powerful.
Google is investing in ARM, Facebook is investing in ARM, and I can only imagine several other big players are looking into it too. I wouldn't write it off this easily.