Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

61–70 of 268 posts

Re: Slashing data transfer costs in AWS

#61

Post by whstl »

maeln wrote: post #10
andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

> Seriously, if you’re at the point that you’re doing sophisticated analysis of cloud costs, consider dropping the cloud. Which would mean that you loose part of the reason to use the cloud in the first place... A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control (amongst many other thing). This can make a lot of sense depending on your infra, if you have some fl…

There is a massive difference between "[if X happens] consider dropping the cloud" and "cloud bad".

Re: Slashing data transfer costs in AWS

#62

Post by steveBK123 »

sofixa wrote: post #36
ownagefool wrote: post #25

Earlier quoted context omitted.

> A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control I think a lot of orgs move to cloud simply because it's popular and gartner told them so. But taking a step away from that, it's really about self-service. When the alternative is logging a ticket for someone to manually misconfigure a VM and then fail to send you the login credentials, then your delivery is s…

> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…

If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team.

A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which could take a day to a week. This was not a temporary state of affairs either, it was allowed to run like this for 2 years+.

Oh and, worth mentioning, CTO there LOVED him some Gartner.

Re: Slashing data transfer costs in AWS

#63

Post by whstl »

robertlagrant wrote: post #14
andrewstuart wrote: post #11

Earlier quoted context omitted.

>> This whole "cloud bad" is just childish. Not childish…. it’s a growing line of thought in the IT community that has bought the cloud sell unquestioningly for 20 years.

But it's the same mindset as "cloud good", which was also a growing line of thought once. Mantras aren't useful; tradeoff analysis is useful.

The GP said "[if X happens] consider dropping the cloud". Which is totally different from a mantra.

There is virtually nobody saying "cloud bad" without nuance.

Re: Slashing data transfer costs in AWS

#64

Post by steveBK123 »

mcny wrote: post #21
robertlagrant wrote: post #14

Earlier quoted context omitted.

But it's the same mindset as "cloud good", which was also a growing line of thought once. Mantras aren't useful; tradeoff analysis is useful.

Mantras are good for orgs that are not mature enough to do actual analysis. A lead developer left recently where I work and while it was likely higher pay that was the biggest decision to move, I suspect the real reason he left is higher ups simply don't listen when he says things like you can't just take full virtual machines on azure, refuse any rewrite/redesign while complaining about high azure spend.

AWS is infamous in financial services for this though.

First they give you a ton of credits, assign you internal resources to help.

Then they encourage you to simply "lift and shift" your workloads onto EC2/EBS/EFS/etc. It's 100% compatible with your current system, you can rollback, etc. This take two years, then you notice your AWS bill is 10x your old infra.

Then they say - of course, that's because you need to rewrite it all to serverless/microservices/etc that are all AWS bespoke branded alphabet soup of services. Now you are fully entrapped, and can not rollback to your own infra, let alone another cloud provider without another rewrite.

A lot of big financial firms are 5+ years into this. Several have rolled back for certain use cases due to cost, especially anything with a lot of data transfer because yeah.. performant storage in the cloud & egress are expensive, duh.

Re: Slashing data transfer costs in AWS

#65

Post by photonthug »

ownagefool wrote: post #25
maeln wrote: post #10

Earlier quoted context omitted.

> Seriously, if you’re at the point that you’re doing sophisticated analysis of cloud costs, consider dropping the cloud. Which would mean that you loose part of the reason to use the cloud in the first place... A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control (amongst many other thing). This can make a lot of sense depending on your infra, if you have some fl…

> A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control I think a lot of orgs move to cloud simply because it's popular and gartner told them so. But taking a step away from that, it's really about self-service. When the alternative is logging a ticket for someone to manually misconfigure a VM and then fail to send you the login credentials, then your delivery is s…

> I think a lot of orgs move to cloud simply because it's popular

This can be rational and not just following the leader. In particular.. many devs might think that working with an org that does On-prem is bad for their career, and they might be right. So from an org POV you can't hire good engineers if you're perceived as a dinosaur. This actually might be enough to send you towards the cloud even if the price by itself makes no sense

Re: Slashing data transfer costs in AWS

#66

Post by robertlagrant »

steveBK123 wrote: post #64
mcny wrote: post #21

Earlier quoted context omitted.

Mantras are good for orgs that are not mature enough to do actual analysis. A lead developer left recently where I work and while it was likely higher pay that was the biggest decision to move, I suspect the real reason he left is higher ups simply don't listen when he says things like you can't just take full virtual machines on azure, refuse any rewrite/redesign while complaining about high azure spend.

AWS is infamous in financial services for this though. First they give you a ton of credits, assign you internal resources to help. Then they encourage you to simply "lift and shift" your workloads onto EC2/EBS/EFS/etc. It's 100% compatible with your current system, you can rollback, etc. This take two years, then you notice your AWS bill is 10x your old infra. Then they say - of course, that's because you need to re…

You can still use standard stuff like Kubernetes, even if you go microservices. I don't think it's that bad.

I'd say Cloud lets you do a few things, but the way I think of it ultimately is it lets you spend opex instead of capex. If that means though that your opex will end up higher than your capex, then it would be silly to go with it.

The other thing is in theory your reliability should be higher, but, again, that will depend on your individual situation, and how much reliability matters to you.

Re: Slashing data transfer costs in AWS

#67

Post by TheNewsIsHere »

This may be arguably nitpicking, but the following statement from TFA isn’t exactly the case:

> Moreover, uploading to S3 - in any storage class - is also free!

Depending upon how much data you’re transferring in terms of storage class, number of API calls your software makes to do so, and the capacity used, you may incur charges. This is very easy to inadvertently do when uploading large volumes of archival data directly to the S3 Glacier tiers. You absolutely will pay if you end up using millions of API calls to upload tens of millions of objects comprising tens of terabytes or more.

Re: Slashing data transfer costs in AWS

#68

Post by marcus0x62 »

steveBK123 wrote: post #62
sofixa wrote: post #36

Earlier quoted context omitted.

> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…

If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team. A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which coul…

I've never seen an IT team that couldn't spin up a VM in minutes. I have seen a bunch of teams that weren't allowed to because of ludicrous "change control" practices. Fire the managers that create this state of affairs, not the devops folks, regardless of whether you "go cloud" or not.

Re: Slashing data transfer costs in AWS

#69

Post by danielklnstein »

TheNewsIsHere wrote: post #67

This may be arguably nitpicking, but the following statement from TFA isn’t exactly the case: > Moreover, uploading to S3 - in any storage class - is also free! Depending upon how much data you’re transferring in terms of storage class, number of API calls your software makes to do so, and the capacity used, you may incur charges. This is very easy to inadvertently do when uploading large volumes of archival data dir…

Thanks for the feedback! I don't think it's nitpicking, you're right that it's misleadingly phrased - in fact, the only S3 costs I observed weren't storage at all, but rather the API calls.

I updated the phrasing.

Re: Slashing data transfer costs in AWS

#70

Post by mlhpdx »

andrewstuart wrote: post #11
maeln wrote: post #10

Earlier quoted context omitted.

> Seriously, if you’re at the point that you’re doing sophisticated analysis of cloud costs, consider dropping the cloud. Which would mean that you loose part of the reason to use the cloud in the first place... A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control (amongst many other thing). This can make a lot of sense depending on your infra, if you have some fl…

>> This whole "cloud bad" is just childish. Not childish…. it’s a growing line of thought in the IT community that has bought the cloud sell unquestioningly for 20 years.

Oh how I wish it had been so. The cloud has been a hard sell all along. Also, 20 years ago S3 and EC2 didn’t exist, so maybe it’s been a little less time than that.
Post reply on HN