Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

121–130 of 268 posts

Re: Slashing data transfer costs in AWS

#121

Post by perryizgr8 »

andersa wrote: post #24

I don't understand how AWS can keep ripping people off with these absurd data transfer fees, when there is Cloudflare R2 just right over there offering a 100 times better deal.

When all my VMs and containers are hosted in AWS, and S3 has rock solid support no matter what language, framework, setup I use, it becomes really tough to ask the team to use another vendor for object storage. If something goes wrong with R2 (data loss, slow transfer, etc.) I will get blamed (or at least asked for help). If S3 loses data or performs slowly in some case, people will just figure we're somehow using it wrong. And they will figure out how to make it better. Nobody gets blamed. And to be honest, data transfer fees is negligible if your business is creating any sort of value. You don't need to optimise it.

Re: Slashing data transfer costs in AWS

#122

Post by declan_roberts »

jonatron wrote: post #16

If you're a heavy bandwidth user it's worth looking at Leaseweb, PhoenixNAP, Hetzner, OVH, and others who have ridiculously cheaper bandwidth pricing. I remember a bizarre situation where the AWS sales guys wouldn't budge on bandwidth pricing even though the company wouldn't be viable at the standard prices.

That’s very unusual, I think. Transfer cost seem to be something most people can negotiate.

Re: Slashing data transfer costs in AWS

#124

Post by huslage »

lijok wrote: post #119

There are tons of these tricks you can use to cut costs and get resources for free. It's smart, but not reliable. It's the same type of hacking that leads to crypto mining on github actions via OSS repos. Treat this as an interesting hacking exercise, but do not deploy a solution like this to production (or at least get your account managers blessing first), lest you risk waking up to a terminated AWS account.

I have used this and other techniques for years and never gotten shut down. Passing through S3 is also generally more efficient for distributing data to multiple sources than running some sync process.

Re: Slashing data transfer costs in AWS

#125

Post by shermantanktop »

steveBK123 wrote: post #99
sofixa wrote: post #83

Earlier quoted context omitted.

I've met multiple customers where time to get a VM was in the weeks to months. (To be fair, I'm at a vendor that proposed IaC tooling and general workflows and practices to move away from old school ClickOps ticket-based provisioning, so of course we'd get those types of orgs). And more often than not, it had nothing to do with managers, but with individual contributors resisting change because they were set in their…

I think a lot of HN crowd hangs out at FAANG/FAANG adjacent or at least young/lean shops, and has no idea how insane it is out there. I was at a shop that provisions AWS resources via written email requests & clickops, treated fairly similar to a datacenter procurement. Teams don't have access to the AWS console, cannot spin up/down, stop, delete, etc resources. A year later I found out that all the stuff they provis…

I went from that type of world (cell carrier) to a FAANG type company and it was shocking. The baseline trust that engineers were given by default was refreshing and actually a bit scary.

I’m not sure my former coworkers would have done well in an environment with so few constraints. Many of them had grown accustomed to (and been rewarded and praised for) only taking actions that would survive bureaucratic process, or fly underneath it.

Re: Slashing data transfer costs in AWS

#126

Post by Sebb767 »

dijit wrote: post #9
cdchn wrote: post #5

Earlier quoted context omitted.

Which VPS services don't charge you at all for bandwidth?

you'd be harder pressed finding ones that do (before a reasonable limit). tilaa.com vultr.com hetzner.com linode.com

> before a reasonable limit

Which is another way of saying you buy a lump sum of bandwidth included with your VM.

The issue is not paying for bandwidth, the issue is the insane pricing.

Re: Slashing data transfer costs in AWS

#127

Post by shermantanktop »

Radim wrote: post #85
photonthug wrote: post #65

Earlier quoted context omitted.

> I think a lot of orgs move to cloud simply because it's popular This can be rational and not just following the leader. In particular.. many devs might think that working with an org that does On-prem is bad for their career, and they might be right. So from an org POV you can't hire good engineers if you're perceived as a dinosaur. This actually might be enough to send you towards the cloud even if the price by it…

I've experienced the opposite too: orgs looking at down on cloud-only devs. The idea being that devs who lean on cloud excessively do that to masks their lack of fundamentals, which will cause costly fuck ups no matter what technology they use, cloud or on-prem. Maybe directionally similar idea to hiring ex-Googlers? Some orgs also don't like those. Specific mindset, specific toolbox.

It is absolutely true that some devs have the AWS product set as the tech toolkit they know best.

Whatever their fundamental skills are, the most important way they add value is by optimizing things like lambda startup time or EC2 CPU utilization. Does this allow them to mask deep problems with fundamentals? I guess it could, but that sounds a bit gatekeep-y to me.

Re: Slashing data transfer costs in AWS

#128

Post by kevin_nisbet »

echelon wrote: post #105
overstay8930 wrote: post #91

Earlier quoted context omitted.

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

Managing on-prem hardware may not necessarily be hard, but it can be extremely time consuming. To me, the nice thing about dropping a bunch of hardware in a colo is you get to take a lot of shortcuts and take risks that you cannot buy from the public cloud providers.

I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service.

I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.

Re: Slashing data transfer costs in AWS

#129

Post by mdale »

lijok wrote: post #109
ttul wrote: post #107

Earlier quoted context omitted.

Incompetence. Take my friend’s company for instance. They were frustrated paying $60K/mo to Amazon so their brilliant sysadmin bought $600K of servers and moved them into a cheap colo. Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result. The sysadmi…

No key person risk management -> no risk register -> no management. Your friends company will fail regardless of poor sysadmin decision making or not. They need to hire competent management ASAP.

With cloud and SaaS services you are paying to reduce person risk profile.

Your forming a larger dependency on a team lead against a custom system that now is a liability as new people come to the organization don't want to adopt an abandoned poorly understood project.

Re: Slashing data transfer costs in AWS

#130

Post by glintik »

holoduke wrote: post #111
overstay8930 wrote: post #91

Earlier quoted context omitted.

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…

My small business once spend 50k per month on AWS. We brought that back to 800 dollars for a similar setup at Hetzner. I find this a significant number.

Reliability differs, right?
Post reply on HN