Why cloud bandwidth is so obscenely expensive and what you can do about that

I think they do a really good job of it, personally. It's trivial to get your free account set up to play with and even host a pretty serious amount of "simple" traffic, and you can easily explore the features, see what's available for paid (they're quite transparent about the assorted costs which is nice), etc.

It feels a bit like an honest version of Microsoft and Adobe's "look the other way for casual piracy, because you're familiar with the product..." of the late 90s and early 2000s. "Hey, try it out for free, and if the free stuff works for you, great. But if you need more, this is what else we offer!"

This is obviously false to me as someone who has done data center stuff and devops before.

The hardest part is running complex services like databases, storage, and runtime environments like Kubernetes or serverless and doing a good enough job to deliver high uptime for a gigantic variety of customer workloads over which you have basically no control.

The other hardest problem is security, because you are by definition running a ton of untrusted code on shared infrastructure. (scream emoji!)

You can sort of tell software management is where the pain is because that's what keeps people on the cloud. That's the selling point. Without those managed services compute, storage, and bandwidth are ridiculously cheaper on VPS and bare metal providers.

Networking is comparatively easy. It's mostly a game of over-engineering. You design your internal data center fabric to handle way more data than you think you will need.

There was a time maybe a decade or two ago when networking at scale was a lot harder but today there are numerous off the shelf products designed for hyper-scale that you can just buy and deploy. This is part of why VPS and bare metal providers can sell bandwidth so cheap. You don't have to roll your own high performance fiber backplane fabric anymore.

This doesn't really make sense as an explanation.

Cloud networking has two components:

1. The network connecting the datacentres to the internet (the datacentre edge infrastructure).

2. The network connecting the servers and virtual machines to that edge.

1 is a solved problem and the variable costs are going to be based on any transit connections used. What cloud providers charge for bandwidth here is out of proportion to that, and they don't bill for ingress anyway, so this makes no sense.

2 is a technically more complex area, except it's very clear that getting people to use this infrastructure less is not the motivation because cloud providers don't charge for intra-DC traffic. You can ferry data around between servers in the same AZ as much as you like and not pay anything. The same is true of transferring data to and from e.g. S3 and EC2.

In other words, the more technically complex and sophisticated network component is the one which isn't billed at all. Meanwhile, boring internet connectivity is billed at an exorbitant rate, but only in one direction.

It's really quite obvious at this point that the motivation is vendor lock-in. You are penalised if you take data out of a big cloud provider but not to put data in. This means as soon as you put anything in AWS you are suddenly motivated to do as much in AWS as possible.

This is fundamentally anti-competitive. If AWS starts offering some infrastructure service, it can offer free bandwidth to that service for its own customers, but some third party providing a comparable service can't, at least unless it also starts using AWS.

But that's not even the best of it. If you want even better proof that these charges are BS, just look at Snowball. This is a service where you can transfer data to S3 by having AWS mail you a hard drive. Amazingly, they have the gall to charge per GB for this in addition to the fixed fee you pay... but only for exporting data.

In other words, I can pay $150 and get loaned a SSD, fill it up with 14TB of data and import that data to S3. If I pay $300 to get data out of S3, I also pay $0.03/GB. Internet transit is not even involved here!

It is pure anti-competitive lock-in. It's also really against the principles of the internet, in that the internet couldn't have developed as it did if we accepted different billing for communicating with different IPs (can you imagine "I don't want to send traffic to a foreign IP, I'll pay international rates!"). Yet cloud providers work on this principle. If you think about it it is effectively a net neutrality violation, just not on the part of residential ISPs - they charge you less to communicate with their own services than with others (zero-rating).