Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud Egress Costs

getdeploying.com

Post reply on HN
Print view View original

281–290 of 318 posts

Re: Cloud Egress Costs

#281

Post by ehhthing »

Nevermark wrote: post #264
ehhthing wrote: post #212

Earlier quoted context omitted.

Ingress is free because it helps them balance their pipes, and it would be really shitty to charge for DDoS attacks. As far as I can tell, with the exception of some really expensive network environments (e.g. China), nobody has ever charged for ingress. With an exception to OVH, none of the cheap providers the article has listed have any kind of backbone network. They all rely fully on transit providers. Turns out b…

> Ingress is free because it helps them balance their pipes That isn't how business works. Companies maximize their profits and "balance" isn't a profit center. If it didn't benefit them in some customer leveraging way, they would charge for ingress. You pay for everything. Either directly or indirectly. Indirectly often turns out to be much more expensive.

> That isn't how business works. Companies maximize their profits and "balance" isn't a profit center. If it didn't benefit them in some customer leveraging way, they would charge for ingress.

What I'm referring to is the practice of balancing peering ratios. That is, when you make transit/peering arrangements with other ISPs, some ISPs will charge more if the amount of data you're sending to them vs the amount of data you're receiving from them is not balanced. It is in Google's best financial interest to at least try to balance their pipes in this way.

Re: Cloud Egress Costs

#282

Post by Fripplebubby »

rambambram wrote: post #241

Let's say you pay a lot for some cloud provider and you find out that your app is best and cheapest served by a good ol' regular webhosting provider. Please, in that case, make sure you leave your money and data at independent webhosts. There's a trend going on - at least in Europe - of consolidating smaller webhosts into big players, under the guidance of some big groups (private equity?). They quickly raise their p…

If you believe that egress prices are fake and pure profit, I encourage you (or someone else reading this with the same belief) to try running your own independent web hosting service ;)

I think what you will discover is that network bandwidth is a finite resource you have to pay for and ration out to your customers, and that the easiest way to finitize and ration something your customers expect (errantly) to be infinite is to charge for it, and charge exactly that price that will cap demand!

Re: Cloud Egress Costs

#283

Post by ehhthing »

Scubabear68 wrote: post #280
ehhthing wrote: post #212

Earlier quoted context omitted.

Ingress is free because it helps them balance their pipes, and it would be really shitty to charge for DDoS attacks. As far as I can tell, with the exception of some really expensive network environments (e.g. China), nobody has ever charged for ingress. With an exception to OVH, none of the cheap providers the article has listed have any kind of backbone network. They all rely fully on transit providers. Turns out b…

Again with the word salad. It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform. They therefore charge high fees to move your data off the platform to discourage this behavior. Meaning you now need to use cloud provider X’s services to do anything with the data. Attempts at vendor lock-in have been core to soft…

> It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform.

Right but if this were the case then why does the Bandwidth Alliance allow you to move data at a much lower cost for 2/3 of the major cloud providers? If they _really_ cared so much about not allowing you to do processing with a third party, the Bandwidth Alliance wouldn't exist!

AWS is the sole hold-out here, and I think the way that Cloudflare worded this makes it pretty clear that the Bandwidth Alliance is basically a middle finger to AWS than anything else, but it also seems clear that the cloud companies aren't actively trying to make it costly to do data processing on a third party.

In fact if you want to move off GCP right now, Google will waive all egress fees to do so: https://cloud.google.com/blog/products/networking/eliminatin...

Re: Cloud Egress Costs

#284

Post by candiodari »

vidarh wrote: post #270
lumost wrote: post #263

Earlier quoted context omitted.

It’s not so much the setup time for the hardware/networking. This takes some time, but can reasonably be done in a few months even for complex setups. The problem comes when you then wish to have things like CI/CD, distributed databases etc.

The vast majority of companies never reach a scale where wiring up the infra needs to take more than days. And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it. Done this many times.

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience.

You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point.

2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse.

Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy".

There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari.

3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions.

4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)

Re: Cloud Egress Costs

#285

Post by sgarland »

aflukasz wrote: post #229
sgarland wrote: post #88

Earlier quoted context omitted.

The inter-AZ charges kill me. They’re charging you for traffic over what is almost certainly private fiber, of a distance no more than 50 miles or so. Just absurd. Then, they have the audacity to recommend you make everything Multi-AZ. “Don’t worry, RDS-RDS traffic isn’t billed!” Yes, but the apps running in us-east-1a that are hitting -1b and -1c due to load balancers most assuredly do bill for traffic.

BTW, if nothing changed the last year or two, then even if you want to run in a single AZ, but also to have (automatically) replicated RDS instance, then I believe the replica MUST be in another AZ. Meaning that any potential failover puts you in exactly that position you have mentioned - apps running in one AZ, database in the other. Suddenly you pay 0.01USD/GB for transfer between those, billed in each direction se…

Correct. The same thing also happens if you use the Blue/Green functionality to do an upgrade – unless you specify (and even then, I'm not sure if you can do so), the Blue instance will be in a different AZ than Green.

Re: Cloud Egress Costs

#286

Post by ren_engineer »

amluto wrote: post #41
PreInternet01 wrote: post #7

The cost of egress traffic is a very good reason for many organizations to not fully migrate to a cloud provider anytime soon. And since, unlike with storage costs, there doesn't seem to be an actual reason (other than: it makes migrating to competitors cost-prohibitive in a subset of cases), that seems kind of... weird? Small example: an actual company I do some work for is in the business of delivering creative ass…

> other than: it makes migrating to competitors cost-prohibitive in a subset of cases My theory: it forces third party services into the same cloud. Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples: https://www.snowflake.com/en/data-cloud/pricing-options/ And look at this remarkable choice: you get to pick AWS…

Snowflake and most other cloud services offer on-prem for these use cases, so it really doesn't make sense for them to roll their own data centers because it would be pretty niche. Cloud works for startups because they are there already, on-prem for enterprise customers with their own hardware

Re: Cloud Egress Costs

#287

Post by lumost »

candiodari wrote: post #284
vidarh wrote: post #270

Earlier quoted context omitted.

The vast majority of companies never reach a scale where wiring up the infra needs to take more than days. And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it. Done this many times.

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sa…

Once in my career, I did a high 7 figure migration from cloud to colo.

Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud.

Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort.

Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste.

2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%.

By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI.

Given this backdrop, I never worked on colos again (this all went down 10+ years ago).

Re: Cloud Egress Costs

#288

Post by creativeSlumber »

zer00eyz wrote: post #214
quickthrower2 wrote: post #161

Earlier quoted context omitted.

So an anticompetitive practice.

No. Everyone knew what the costs were going in. AWS, cloud was NEVER the cheaper solution. It was never faster, it was never better. It was easier, and easier came with a price. Like lambs to the slaughter lots of people embraced it. Amazon is a profitable company because of it.

> Everyone knew what the costs were going in.

Knowing the cost beforehand and anti-competitive are two different things.

Re: Cloud Egress Costs

#289

Post by rstupek »

close04 wrote: post #269
growse wrote: post #258

Earlier quoted context omitted.

I'm a little ignorant of the upcoming regs, but are they aiming to basically say "if a customer wants to leave a service, then the provider must provide them with all their data for free"? I've not thought about the unintended consequences of this, but it feels like a reasonable regulation to have.

Ingress and egress should be treated the same. Without any anti-competitive reasoning in mind a provider can’t claim that egress in particular should be more expensive than ingress. Even more so when ingress is many times completely free. The asymmetry is obviously meant to trap customers, which is anti-competitive.

I'm pretty sure ingress started off costing less due to the nature of usage. If you were to see a graph of the usage you'd see ingress is used less. One reason is an http request takes less bytes than the response to the request. So the disparity in cost can't simple be attributed to a desire to trap customers

Re: Cloud Egress Costs

#290

Post by vidarh »

lumost wrote: post #287
candiodari wrote: post #284

Earlier quoted context omitted.

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience. You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sa…

Once in my career, I did a high 7 figure migration from cloud to colo. Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud. Everything went smooth enough at first, the ha…

So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason?

Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed.

1. Don't do migrations at once. In a large deployment you likely have multiple locations anyway, so do one at a time, and/or rack by rack.

2. Verify usage levels, and take advantage of your ability to scale into cloud instances to ensure you only provision tiny headroom on your non-cloud environment. Doing so makes the cost gap to a cloud environment much bigger.

3. The advantage is being able to tune servers to your workload. That you had to fight for SSDs is already a big red flag - if you don't have a culture of testing and picking based on what actually work, yeah, you may actually be best off in the cloud because you'll overpay by a massive factor no matter what you choose and in a cloud you can at least rapidly rectify the repeated major mistakes that will be made.

4. Don't pay upfront unless you're at a scale where it brings you discounts at a return that makes it worth it. Rent or lease to own. For smaller environments, rent managed servers. The cloud provider dropping prices 2 years in shouldn't matter because by year 3 you should have paid off the hardware at an average cost well below anyway, and start cycling out hardware as your load requires and as/when hardware fails and/or it's not cost effective to have it take up space in your racks any more. Depending on your setup, that might be soon, or you might find yourself still running 6+ year old hardware you paid off 3 years ago for many tasks.

If you can't beat cloud on TCO, you're doing something very wrong. I use cloud for lots of things that aren't cost sensitive, but there are extremely few scenarios where it is the cost effective option.

Post reply on HN