Live data from Hacker News

Cloud Egress Costs

getdeploying.com

291–300 of 318 posts

Re: Cloud Egress Costs

#291
post #270

Earlier quoted context omitted.

The vast majority of companies never reach a scale where wiring up the infra needs to take more than days. And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it. Done this many times.

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sa…

I wrote an orchestrator from scratch in the pre-K8S days, and even factoring in the opportunity cost of that colo and eventually Hetzner was far cheaper than cloud, to the point where - and we costed this out yearly - the company wouldn't have survived a year if we had to pay AWS level prices for infrastructure.

Cloud is great for many things, but not cutting cost.

Re: Cloud Egress Costs

#292

Earlier quoted context omitted.

Again with the word salad. It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform. They therefore charge high fees to move your data off the platform to discourage this behavior. Meaning you now need to use cloud provider X’s services to do anything with the data. Attempts at vendor lock-in have been core to soft…

> It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform. Right but if this were the case then why does the Bandwidth Alliance allow you to move data at a much lower cost for 2/3 of the major cloud providers? If they _really_ cared so much about not allowing you to do processing with a third party, the Bandwidth…

Bandwidth alliance looks to be a political tool for cloud providers to save face. Not dissimilar to public companies paying token tribute to ESG which is all the rage these days.

Re: Cloud Egress Costs

#293
Having seen the other side of the fence (the hyperscaler side) I’m kind of bored with egress cost being continuously compared with your standard bare-metal hosting provider for a few reasons:

- Let’s first dispense with the idea that egress has to be provided without profit/margin in a capitalistic society. There will be profit in it sure and I don’t dismiss the idea that egress pricing is used to keep activities on-platform, not that I’ve been part of the decisions to set the price that way.

- Typically the more basic a network is, the easier it is to provision, manage and scale. Having a single DC with a couple of local transit providers and BGP routing brings with it a wildly lower cost base compared with a global network with piles of different POPs.

- Many providers, by charging only by usage, are effectively saying that the network is infinite in capacity and “just works”. You would be surprised how many engineers believe this to be true as well. To that end, as the complexity of the network grows you need to charge in a way that allows you to keep capacity ahead of demand for every path you manage. And then you need geographic redundancy for every such egress path and systems/people to manage said failover.

- In the case of GCP Premium tier, Google is hauling your traffic as far as it can on its private network towards its destination before exiting via a POP. Usage forecasting and pricing as a result needs to effectively assume any VM wants to send any amount of traffic to anywhere in the world. Even then the premium tier pricing separates kit China and Australia as special cases.

- In the hyperscaler case and even many of the larger VM/bare metal hosts you’ll find software defined networks which can often have a non-zero per-byte CPU processing cost involved. AFAIK this is essentially written off when traffic is in the same zone or region but escalates for various reasons (say rate limiting, egress path calculations, NAT, DoS prevention) when going inter-region or to the internet.

- Many of the hyperscalers do allow you spin up private interconnects but often charge for usage across it. This shifts away from being raw cross-connect cost to being more enterprise-y where the value of having dedicated, private capacity becomes the price. There is also the cost of managing said interconnect since it most certainly doesn’t get handled the same way as other egress paths (thus is more of an exception and exceptions cost money/time/effort).

Do all of these things add up to the “high” egress costs plus a decent margin for that evil profit? That is mostly up to the reader and what they value. Many others will say they don’t need all these features/management, but the reality is the hyperscalers don’t build for you, they build to cater to everyone that might become or is a customer. And it turns out to build a network capable of handling what “everyone” could potentially do with it is expensive.

Re: Cloud Egress Costs

#294
post #290
post #287

Earlier quoted context omitted.

Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the ha…

So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason? Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed. 1. Don't do migrations at once. In a large deployment you likel…

Hardware beyond 3 years starts to be inefficient due to the space and power cost. paid off servers aren't free :) Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al.

On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly, but we were in the realm of "we will have to hire people to maintain this project at 24/7 uptime". As the people time was relatively fixed to manage a sane on-call rotation - shrinking the infra footprint would have simply increased the people time portion of the TCO. If the footprint had shrunk by 75% - the maintenance cost would have become problematic. Due to the higher intensity work of build outs vs. maintenance, the people time factor would have risen had we started in more experimental amounts. As it happened, tech salaries also rose ~2x over the 3-5 years that Colo existed.

On the workload front, as happens with many large organizations - there are teams who have the attitude of "don't touch my stuff". At the time that the buy was initiated, there were several estimates indicating that we would need 4x the hardware we were buying in 5 years time.

Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math.

- Sum up all costs related to the on-prem hardware, include smart-hands, power, Racks/network gear, power adaptors.

- Calculate a depreciation schedule for the existing hardware targeted to 3 years.

- Add a 10% cost of capital to depreciate future savings

- Amortize all "non-server" hardware costs onto your principal bottleneck (be it CPU/Network/Memory/Storage)

- Project how your savings compare against cloud costs under different assumptions of future cloud discounts. Both negotiated and public.

- Project how your savings compare under different utilization assumptions.

- Add a bus factor into the investment, what happens if your team leaves/hardware gets smashed/Workload becomes more efficient/Other crises occurs

Next, and more controversially

- Add in the people cost of everyone involved in the maintenance of your on-prem infrastructure.

- Project what happens under varying assumptions for how much of a raise they will ask for. What happens if tech compensation rises by another 2x over the next 3 years?

- Explore what happens if on cloud you needed X% fewer engineers. Depending on what you are doing X could be 0%, or even a negative percentage - but for many shops some fraction of existing work can be automated or avoided.

It was this math that made me turn away from non-cloud offerings. I still use cloud alternatives for some personal projects - but I don't bill myself in that setting.

Re: Cloud Egress Costs

#295
post #109

Earlier quoted context omitted.

Context matters here. How critical is that workload? What the economic and reputational impact for the company, if one of the physical connections or some technical problem with the data center causes a downtime of hours or days?

looking at the uptime from aws and from most big outtage notices i have read in the last few years, there does not seem to be a benefit in regards to reliability when using cloud. see reddit, see amazon/aws outages taking with them netflix/disney plus etc honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region also: store your backups somewhere els…

I am not aware of an AWS outage in their 15-16 years of existence, that an architecture, built according to the recommended best practices, of distribution across availability zones and regions, would not be able to withstand. I am willing to be proven wrong. Can you provide one example?

Of course, these come with increased cost, but I am thinking web retail on a large scale, or airline companies, where for example a downtime of a few hours will easily wipe out any savings made by relying on a local data center. It might not be the solution for a smaller company.

Re: Cloud Egress Costs

#296
post #258

Earlier quoted context omitted.

I'm a little ignorant of the upcoming regs, but are they aiming to basically say "if a customer wants to leave a service, then the provider must provide them with all their data for free"? I've not thought about the unintended consequences of this, but it feels like a reasonable regulation to have.

Ingress and egress should be treated the same. Without any anti-competitive reasoning in mind a provider can’t claim that egress in particular should be more expensive than ingress. Even more so when ingress is many times completely free. The asymmetry is obviously meant to trap customers, which is anti-competitive.

Wholesale connectivity is usually priced per unit for the 95th percentile sample, in the dominant direction. For most cloud services, outbound is the dominant direction by far. That's why ingress is free in almost every hosting environment.

Additionally, settlement free peering is usually based on having a roughly balanced traffic ratio (something like 2:1 or 3:1 counts as roughly balanced), attracting more ingress traffic by making it free vs having paid egress helps the provider balance their ratios so they may help their case for settlement free peering.

Re: Cloud Egress Costs

#297

The article says “ Cloud providers charge for egress because it costs them money to send data out of their network. They have to pay for the infrastructure and bandwidth required to send data to users”. The charge is not based on cost in the case of the big names. They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free. Hence the high costs to…

> They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free. This isn't remotely true. The bandwidth alliance exists, and a lot of cloud companies are on the list: https://www.cloudflare.com/en-gb/bandwidth-alliance/ The actual answer is much more complicated. For example, Google Cloud offers two different bandwidth tiers: premium and standard.…

> For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive.

Of course, non-premium tier is v4 only, and only available at some locations.

Re: Cloud Egress Costs

#298
post #207

Earlier quoted context omitted.

What’s extra special is that the cloud providers are now disincentivised from making those load balancers zone topology aware. If they made them efficiently pick the same zone whenever possible, they would be missing out on cross-AZ traffic billing, which is basically a money printer.

AWS ELB certainly is zone aware, iirc cross-zone load-balancing is opt-in

Azure load balancers certainly aren’t.

I guess that’s just one more reason to prefer AWS over Azure.

Re: Cloud Egress Costs

#299
post #294
post #290

Earlier quoted context omitted.

So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason? Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed. 1. Don't do migrations at once. In a large deployment you likel…

Hardware beyond 3 years starts to be inefficient due to the space and power cost. paid off servers aren't free :) Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al. On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I…

> Hardware beyond 3 years starts to be inefficient due to the space and power cost. paid off servers aren't free

Hence my caveat, but racks are typically rented at a fixed price per rack, and most places never even reach a scale where they need a full rack per location, so my experience is that for most people it pays to keep servers quite a bit longer because most people have spare space in racks that are already being paid for.

Once you're at a scale where you typically will have whole racks aging out at once, it shifts the calculation somewhat, but then to it really varies greatly depending on e.g. your balance between storage and compute. It's very rare it pays to throw out hardware on the 3 year mark, except in markets where the power and real-estate cost is unusually high, but then moving your hosting wholesale often pays off - e.g. I've in the past moved entirely workloads from London to cheaper locations.

> Otherwise, we did mortgage the servers in the backend - buying upfront gave us around a 50% discount on top of the steep discounts you get from buying direct from an integrator vs. Dell et. al.

Either you loan-financed the hardware or the earlier mention of the "use of VC funds" was irrelevant, then. And nobody gives you a 50% discount for paying up front. You might have paid 50% less than the full cost of the purchase price + interest rate, sure. That's not a discount, that's not having to pay interest. In the end, whether you buy on credit, rent, or lease to own, you either way get a cost-curve per server per month, and that is what is relevant to plug into your models.

> On the rent-to-own topic, the people time cost of the migration was budgeted at around 10-20% of total TCO if I recall correctly

Migration is a one off, so this only makes sense given a time frame to write it off over. 10%-20% written off over 1-2 years wouldn't be completely crazy, though high. If your time horizon is so short that this matters to you, then you have organizational problems.

Put another way: While I did consulting in this space, I'd often offer to do the transition for clients for a percentage of their savings for the first few months, because I knew exactly how much these transitions would take us, and the clients would take a look at the proposals and accept my hourly rate instead when they realised how much they'd save, how quickly.

Reduction in egress fees alone often paid for my fees in couple of months (one system I migrated, which was admittedly atypical, saw hosting costs drop 90% thanks to egress fees alone), and we usually saw devops costs drop at the same time. Most of my clients outsourced 100% of their devops so the costs were easy to quantify.

> If the footprint had shrunk by 75% - the maintenance cost would have become problematic.

This is backward thinking. If the footprint drops by 75%, the cost for that drops. If your maintenance costs don't drop as fast, it doesn't matter - your total cost is still lower, but if your maintenance cost isn't elastic, you have an organizational problem.

And while whether or not the proportion spent on each factor then changes might be a political consideration, but if you then surrender savings because the budget "would have become problematic" then it's no wonder you ended up with a failed transition - if there are incentives to avoid savings to maintain numbers that were broken from the outset. This sounds more and more dysfunctional to me.

> Whenever someone says that they are beating cloud TCO, I'd suggest you do the following math.

Done all of these many times, and never once had cloud come out remotely competitive.

To your "controversial" point, what I usually see when people think their cloud setup is comparative is that they carry out no accounting of how much time they actually spend maintaining cloud-specific things. When they get to the point of handing it over to someone specializing in it (as I did for years), it's often a surprise to them just how much time they offload from their teams.

A few of the other things that seem to shine through here is 1) an assumption of capital outlays. Not needed capital outlays for coloed environments in any setup I've done in the last 20 years - did it for a handful before that; the cost of financing directly with the provider or via a leasing company is priced in when I compare costs with cloud because otherwise it wouldn't comparable. If you then want to pay upfront, that's a choice, not a necessity unless your credit is absolutely worthless.

2) Comparing only colo vs. cloud instead of adding in hybrid or managed hosting. If you build a pure self-honest environment a lot of the price advantage gets eaten up because you need to assume a far higher amount of spare capacity, which will drive up your cost even if you get it right, but people often end up far too conservative here and assume peaks far higher than what they ever need.

The moment you have a hybrid setup that can scale into cloud instances as needed, which is typically little extra effort to set up (you're going to be using an orchestrator anyway), you can easily double (or triple, if people were being conservative) the typical load on your colo servers, and cut the hardware and rack cost accordingly, and usually when people do this they still end up hardly ever actually spinning up cloud instances, because most peoples traffic varies far less than they'd like to think.

Even more so given there are now plenty of providers that offer you a seamless transition from colo, via managed server, to vps's to cloud instances, with often surprisingly little difference in time to spin up extra capacity. Your setup just has a method to register a new resource anyway, irrespective of what is underneath - I've deployed systems that way for nearly 20 years, now, with hybrid setups spanning the gamut from colo, via managed server, VPSs and AWS instances in a single setup.

The net effect tends to be to have to defend retaining the capability to scale into cloud because of how rarely it ends up being used.

3) an assumption that you need to hire people vs. e.g. outsourcing. Most companies never reach a scale where they need even a single full-time person doing hands-on ops - you're better off leaning on colo support, and retainers for monitoring and out-of-hours support for fractional scaling until you reach a scale where staffing several dozen full-time staff becomes viable. I've never had a problem scaling this up/down on an hour-by-hour basis with commitments for base-level needs on a month-by-month basis. For years I used to provide fractional support for companies to facilitate this type of thing.

4) an assumption that you can't automate the same things in a colo environment as in a cloud environment. For a well-managed colo environment, past racking hardware, if you can't boot the system via IPMI etc. straight into an image that automatically enrolls the server in your orchestrator, you're doing something wrong. If your cost of physically managing your servers are more than a rounding error, you''re doing it wrong.

Yet, the flexibility of public cloud environments is like the gold ticket for people doing devops consulting - when I was consulting in this space, the one constant was that the clients in cloud environments ended up paying me 2x-3x as much for assistance for similar size and complexity workloads. And I still usually cut their costs significantly compared to what they used to pay. E.g. the time that goes to maintaining network setups in a typical cloud setup that's solved by plugging things physically into isolated switches is staggering. Yes, usually people could do it cheaper than they are in public cloud setups, but the risk of getting it wrong also tends to be far higher.

Re: Cloud Egress Costs

#300
post #297

Earlier quoted context omitted.

> They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free. This isn't remotely true. The bandwidth alliance exists, and a lot of cloud companies are on the list: https://www.cloudflare.com/en-gb/bandwidth-alliance/ The actual answer is much more complicated. For example, Google Cloud offers two different bandwidth tiers: premium and standard.…

> For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive. Of course, non-premium tier is v4 only, and only available at some locations.

If you're on GCP because you want v6 support you're probably in the wrong place :^)
Post reply on HN