Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

71–80 of 268 posts

Re: Slashing data transfer costs in AWS

#71

Post by photonthug »

steveBK123 wrote: post #62
sofixa wrote: post #36

Earlier quoted context omitted.

> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…

If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team. A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which coul…

Despite years of friendly sounding devops philosophy there's times when devs and ops are fundamentally going to be in conflict. it's sort of a proxy war between devs who understandably dislike red tape and management who loves it, with devops caught in the middle and on the hook for both rapid delivery of infrastructure but also some semblance of governance.

An org with actual governance in place really can't deliver infra rapidly, regardless of whether the underlying stuff is cloud or on prem, because whatever form governance takes in practice it tends to be distributed, i.e. everyone wants to be consulted on everything but they also want their own responsibility/accountability to be to be diluted. Bureaucracy 101..

Devs only see ops taking too long to deliver, but ops is generally frozen waiting on infosec, management approving new costs, data stewards approving new copies across ends, architects who haven't yet considered/approved whatever Outlandish new toys the junior devs have requested, etc etc.

Depends on exactly what you're building but with a competent ops team cloud vs on prem shouldn't change that much. Setting aside the org level externalities mentioned above, developer preference for stuff like certain AWS apis or complex services is the next major issue for declouding. From the ops perspective cloud vs on prem is largely gonna be the same toolkit anyway (helm, terraform, ansible, whatever)

Re: Slashing data transfer costs in AWS

#72

Post by karlkatzke »

lucidguppy wrote: post #38

This feels like the tech equivalent of tax avoidance. If too many people do this - AWS will "just close the loophole". There's not one AWS - there are probably dozens if not hundreds of AWS - each with their own KPIs. One wants to reduce your spend - but not tell you how to really reduce your spend. If you make something complex enough (AWS) - it will be impossible for customers to optimize in any one factor - as eve…

This isn't a loophole. This is by design. AWS wants you to use specific services in specific ways, so they make it really cheap to do so. Using an endpoint for S3 is one of the ways they want you to use the S3 service.

Another example is using CloudFront. AWS wants you to use CloudFront, so they make CloudFront cheaper than other types of data egress.

Re: Slashing data transfer costs in AWS

#73

Post by boiler_up800 »

S3 storage costs are charged per GB month so 1 TB * .023 per GB / 730 hrs per month… should be 3 cents if the data was left in the bucket for an hour.[1]

However sounds like it was deleted almost right away. In that case the charge might be 0.03 / 60 if the data was around for a minute. Normally I would expect AWS to round this up to $0.01..

The TimedByteStorage value from the cost and usage report would be the ultimate determinant here.

[1] https://handbook.vantage.sh/aws/services/s3-pricing/

Re: Slashing data transfer costs in AWS

#74

Post by karlkatzke »

andersa wrote: post #24

I don't understand how AWS can keep ripping people off with these absurd data transfer fees, when there is Cloudflare R2 just right over there offering a 100 times better deal.

Data has "gravity" -- as in, it holds you down to where your data is, and you have to spend money to move it just like you have to spend money to escape gravity.

Re: Slashing data transfer costs in AWS

#75

Post by blantonl »

sgarland wrote: post #53
antonvs wrote: post #35

Earlier quoted context omitted.

> Host it yourself. If you're actually using the features of cloud - i.e. managed services - then this involves building out an IT department with skills that many companies just don't have. And the cost of that will easily negate or exceed any savings, in many cases. That's a big reason that the choice of cloud became such a no brainer for companies.

Next you’ll tell me that full-stack is a lie, and devs don’t actually know how to run a DB.

Maybe I don't want to manage a db, I just want to run one.

Re: Slashing data transfer costs in AWS

#76

Post by Jean-Papoulos »

andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

>Seriously, if you’re at the point that you’re doing sophisticated analysis of cloud costs, consider dropping the cloud.

The blog post's solution is relatively simple to put in place ; if you're already locked-in to AWS, dropping it will cost quite a lot and this might be a great middle ground in some cases.

Re: Slashing data transfer costs in AWS

#77

Post by unclebucknasty »

martinald wrote: post #58
api wrote: post #32

Earlier quoted context omitted.

It’s not a loss leader. Cloud bandwidth pricing is almost pure profit.

It's absolutely amazing that so many devs don't realise this. They seem to think that bandwidth should cost a few cents a month, when in reality it is virtually free. Perhaps the 7c/GB charge was reasonable when AWS came out 15 years ago, but networking has got orders of magnitude cheaper and faster in the intervening time period. What's more odd now that 1gigabit+ home connections are available, it should be obvious…

>it is virtually free

The infrastructure comes at some cost though, right? And there must be some cap on the bandwidth / throughput that a given infrastructure can handle.

So, given these, does it make sense to price bandwidth as a throttle?

Re: Slashing data transfer costs in AWS

#78

Post by steveBK123 »

robertlagrant wrote: post #66
steveBK123 wrote: post #64

Earlier quoted context omitted.

AWS is infamous in financial services for this though. First they give you a ton of credits, assign you internal resources to help. Then they encourage you to simply "lift and shift" your workloads onto EC2/EBS/EFS/etc. It's 100% compatible with your current system, you can rollback, etc. This take two years, then you notice your AWS bill is 10x your old infra. Then they say - of course, that's because you need to re…

You can still use standard stuff like Kubernetes, even if you go microservices. I don't think it's that bad. I'd say Cloud lets you do a few things, but the way I think of it ultimately is it lets you spend opex instead of capex. If that means though that your opex will end up higher than your capex, then it would be silly to go with it. The other thing is in theory your reliability should be higher, but, again, that…

You CAN, but of course that's not what AWS steers you to.

Once your org has gotten to that step, it's been so steered by AWS staff it's hard to imagine suddenly finding sense and building with open standard stuff. Very few AWS shops I have encountered avoid the siren call of various AWS-only or AWS-specific services, which they then become heavily ingrained in..

Generally I do think its mostly about transforming CapEx to OpEx, with the rest of the stuff being noise.

Re: Slashing data transfer costs in AWS

#79

Post by belter »

andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

Those VPS hosting services are a solution for a startup but not to run your internet banking, airline company or public sas product. Plus their infinite egress bandwidth and data transfer claims are only true, as long as you don't...Use infinite egress bandwidth and data transfer....

Re: Slashing data transfer costs in AWS

#80

Post by steveBK123 »

marcus0x62 wrote: post #68
steveBK123 wrote: post #62

Earlier quoted context omitted.

If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team. A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which coul…

I've never seen an IT team that couldn't spin up a VM in minutes. I have seen a bunch of teams that weren't allowed to because of ludicrous "change control" practices. Fire the managers that create this state of affairs, not the devops folks, regardless of whether you "go cloud" or not.

Teams are what they DO, not what they CAN DO.
Post reply on HN