Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

91–100 of 268 posts

Re: Slashing data transfer costs in AWS

#91

Post by overstay8930 »

andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else.

I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go back on prem, or you just don't give a shit about reliability (start pricing up DDoS protection costs at anything higher than 10G and tell me the cloud is more expensive).

We spend 100k+ on AWS bandwidth and it's still cheaper than our DIA circuits because we don't have to pay network engineers to manage 3 different AZs.

Re: Slashing data transfer costs in AWS

#92

Post by steveBK123 »

photonthug wrote: post #71
steveBK123 wrote: post #62

Earlier quoted context omitted.

If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team. A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which coul…

Despite years of friendly sounding devops philosophy there's times when devs and ops are fundamentally going to be in conflict. it's sort of a proxy war between devs who understandably dislike red tape and management who loves it, with devops caught in the middle and on the hook for both rapid delivery of infrastructure but also some semblance of governance. An org with actual governance in place really can't deliver…

Yes, of course management is often the problem.

I think it helps when people actually take a step back and understand where the money that pays their salary comes from. Often times people are so ensconced in their tech bureaucracy they think they are the tail that wags the dog. Sometimes the people that are the most hops from the money are the least aware of this dynamic. Bureaucracies create an internal logic of their own.

If I am writing some internal software for a firm that makes money selling widgets, and I decide that what we really need is a 3 year rewrite of my app for reasons, am probably not helping in the sale or the production of widgets. If another team is provisioning hardware for me to write the software on, and it now takes 2 weeks to provision virtual hardware that could take seconds, then they are also not helping in the sale or the production of widgets.

These are the kind of orgs that someone may one day walk into, blast 30% of the staff, and find no impact on widget production, and obvious 30% savings on widget costs...

Re: Slashing data transfer costs in AWS

#93

Post by lucidguppy »

karlkatzke wrote: post #72
lucidguppy wrote: post #38

This feels like the tech equivalent of tax avoidance. If too many people do this - AWS will "just close the loophole". There's not one AWS - there are probably dozens if not hundreds of AWS - each with their own KPIs. One wants to reduce your spend - but not tell you how to really reduce your spend. If you make something complex enough (AWS) - it will be impossible for customers to optimize in any one factor - as eve…

This isn't a loophole. This is by design. AWS wants you to use specific services in specific ways, so they make it really cheap to do so. Using an endpoint for S3 is one of the ways they want you to use the S3 service. Another example is using CloudFront. AWS wants you to use CloudFront, so they make CloudFront cheaper than other types of data egress.

If they wanted you to behave in specific ways logically - wouldn't their documentation be less ambiguous?

https://www.lastweekinaws.com/blog/aws-cross-az-data-transfe...

Re: Slashing data transfer costs in AWS

#94

Post by amluto »

crabbone wrote: post #44
andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

I don't think "host yourself" in this instance would've helped. I think AWS in this instance is operating at a loss. Author found a loophole in AWS pricing and that's why it's so cheap. Doing it on their own would've been more expensive. Now as to why the AWS pricing the way it is... we may only guess, but likely to promote one service over the other.

AWS is surely operating at a loss for this particular case (zero S3 charge).

But there is no way that cross-AZ traffic costs AWS anywhere near what they charge for it.

Re: Slashing data transfer costs in AWS

#95

Post by barkingcat »

andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

also, if you really need to transfer hundreds of TB's of data from South Africa to the US, put it on a palette of disks and send it via bulk shipping.

then load the data into the datacentre and then just pay for the last sync / delta.

Re: Slashing data transfer costs in AWS

#96

Post by overstay8930 »

rfoo wrote: post #40
develatio wrote: post #26

I'll share my trick :) Lightsail instances can be used to "proxy" data from other AWS resources (eg EC2 instances or S3 buckets). Each Lightsail instance has a certain amount of data transfer included in it's price ($3.5 instance has 1TB, $5 instance has 2TB, $10 instance has 3TB, $20 instance has 4TB, $40 instance has 5TB). The best value (dollar per transferred data) is the $10 instance, which gives you 3TB of traf…

Here's another one: You can download 1TB of data for free from AWS each month, as Cloudfront has a free tier [1] with 1TB monthly egress included. Point it to S3 or whatever HTTP server you want and voila. [1] It used to be 50GB per month for the first 12 months. It was changed to 1TB free forever shortly after Cloudflare posted https://blog.cloudflare.com/aws-egregious-egress

That "shortly" was 2 years, that Cloudflare post had nothing to do with it, Amazon barely considers them a competitor to begin with.

Re: Slashing data transfer costs in AWS

#97

Post by martinald »

unclebucknasty wrote: post #77
martinald wrote: post #58

Earlier quoted context omitted.

It's absolutely amazing that so many devs don't realise this. They seem to think that bandwidth should cost a few cents a month, when in reality it is virtually free. Perhaps the 7c/GB charge was reasonable when AWS came out 15 years ago, but networking has got orders of magnitude cheaper and faster in the intervening time period. What's more odd now that 1gigabit+ home connections are available, it should be obvious…

> it is virtually free The infrastructure comes at some cost though, right? And there must be some cap on the bandwidth / throughput that a given infrastructure can handle. So, given these, does it make sense to price bandwidth as a throttle?

That's why I said 'virtually'.

Hurricane Electric does 40gig/sec IP transit for $2k/month.

Assuming you used 50% of the capacity of that link that's about 1/200th (I think, numbers are so small) of the cost of AWS for bandwidth.

Re: Slashing data transfer costs in AWS

#98

Post by playingalong »

dangoodmanUT wrote: post #39

I've not seen any evidence that multi-AZ is more resilient. There's no history of an entire AZ going down that doesn't affect the entire region, at least that I can find on the internet within 15 minutes of googling.

Do you mean S3 or all services?

If all services, then things like whole or most of a single AZ being borked happens fairly often.

Re: Slashing data transfer costs in AWS

#99

Post by steveBK123 »

sofixa wrote: post #83
marcus0x62 wrote: post #68

Earlier quoted context omitted.

I've never seen an IT team that couldn't spin up a VM in minutes. I have seen a bunch of teams that weren't allowed to because of ludicrous "change control" practices. Fire the managers that create this state of affairs, not the devops folks, regardless of whether you "go cloud" or not.

I've met multiple customers where time to get a VM was in the weeks to months. (To be fair, I'm at a vendor that proposed IaC tooling and general workflows and practices to move away from old school ClickOps ticket-based provisioning, so of course we'd get those types of orgs). And more often than not, it had nothing to do with managers, but with individual contributors resisting change because they were set in their…

I think a lot of HN crowd hangs out at FAANG/FAANG adjacent or at least young/lean shops, and has no idea how insane it is out there.

I was at a shop that provisions AWS resources via written email requests & clickops, treated fairly similar to a datacenter procurement. Teams don't have access to the AWS console, cannot spin up/down, stop, delete, etc resources.

A year later I found out that all the stuff they provisioned wasn't set up as reserved instances. We weren't even asked. So we paid hourly rates for stuff running 24/7/365.

This was apparently the norm in the org. You have to know reserved instances exist, and ask for them.. you may eventually be granted the discount later. I only realized what they had done when they quoted me rates and I was cross checking ec2instances.info I can guarantee you less than 20% of my org (its not a tech shop) is aware this difference exists, let alone that ec2instances.info exists for cross reference.

No big deal, just paying 2x for no reason on already overpriced resources!

Post reply on HN