Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

171–180 of 268 posts

Re: Slashing data transfer costs in AWS

#171

Post by jjav »

overstay8930 wrote: post #143
echelon wrote: post #105

Earlier quoted context omitted.

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

"this is what we're paid for" Nope, it's what YOU'RE paid for. I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed). I have rarely m…

> backwards it is to be anywhere but the cloud

It's almost as if you feel the cloud is something magically different, it's actually just servers on racks owned by someone else.

You can own the same thing if you want and do everything exactly the same.

(See e.g. Oxide)

Re: Slashing data transfer costs in AWS

#172

Post by rented_mule »

no_wizard wrote: post #147
coredog64 wrote: post #141

Earlier quoted context omitted.

Amazon retail runs on AWS, and I think we can agree that Amazon retail is reasonably described as “large scale”

This is a quirk of the business that is Amazon and AWS, as they started by selling excess compute and expertise, due to how Amazon was built as API first internally it was almost natural. It’s in no way the norm for a smaller business.

This matches the public (i.e., non-Amazon) speculation I was hearing around the launch of S3 and later EC2. But not what I was hearing internally when I worked at Amazon. I was there when S3, the first AWS service, and EC2 were launched. I was working on what I believe to be the first Amazon (non-AWS) application that used S3 for storage. Getting that approved was not easy - all the same skepticism existed internally as externally (cost, availability, durability, security, etc.).

The story I was hearing internally was that it was too costly to scale infrastructure the way Amazon had been doing it, it was fragile, and the expertise wasn't keeping up with growth. So, set the bar a lot higher, and build infra that is big enough and flexible enough to be everybody's infra, and then Amazon's applications (e.g., retail) could run on AWS' excess capacity. Literally the opposite of what external folks were guessing. I believe they were completely physically separate data centers - even the physical location of AWS data centers were on a need-to-know basis internally (the internal lore was "under a mountain in Virginia" - this was years before Regions and Availability Zones). And any bugs in AWS could be worked out with outside usage before moving Amazon's applications onto it.

Also, Amazon needed the elasticity of AWS because of the nature of their retail business. At the time that the initial AWS services were being developed, a massive chunk of Amazon's traffic came during the holiday season. IIRC, something like half of the year's traffic and revenue, possibly more, came in November/December each year. That meant a lot of capacity was sitting idle most of the year. Selling that excess capacity would mean shutting AWS down every holiday season.

For a time, there was an internal mailing list that wasn't yet locked down that contained reports on S3 bandwidth usage. The growth rate was shockingly high. I would guess that within a year or two of release, S3 was using a few (at least) orders of magnitude more bandwidth than everything else at Amazon combined.

Re: Slashing data transfer costs in AWS

#173

Post by jjav »

sofixa wrote: post #36
ownagefool wrote: post #25

Earlier quoted context omitted.

> A lot of org move to cloud based hosting because it enable them to go way further in FinOps / cost control I think a lot of orgs move to cloud simply because it's popular and gartner told them so. But taking a step away from that, it's really about self-service. When the alternative is logging a ticket for someone to manually misconfigure a VM and then fail to send you the login credentials, then your delivery is s…

> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…

> waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck..

You still have to go through your devops (or equivalent) team to make any network configuration/permission changes. Whether that change is implemented by a local firewall rule or some AWS configuration change is not very important.

It's not like you're going to have developers changing AWS access permissions directly. Maybe in a few employee startup, but in any regulated & audited company, you must have separation of duties and audited change control process.

Re: Slashing data transfer costs in AWS

#174

Post by 123pie123 »

overstay8930 wrote: post #143
echelon wrote: post #105

Earlier quoted context omitted.

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

"this is what we're paid for" Nope, it's what YOU'RE paid for. I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed). I have rarely m…

there's so much to unpack here!!

Re: Slashing data transfer costs in AWS

#175

Post by mdasen »

slashdev wrote: post #47
develatio wrote: post #33

Earlier quoted context omitted.

A had a suspicion that this was against AWS's terms, but I never bothered to look if that was actually the case. Thank you for the heads up!

It’s mostly in there to scare people into not doing it. AFAIK they’ve never taken action on that. Of course if you abuse it, you’re asking for trouble.

As someone who has dealt with users who use a system in an unintended way, you don't go looking for those people and you don't build something to enforce a policy like this. When you're running services for lots of customers, you often don't know a lot of what's going on in the system and how people are using it. Then something seems weird or something is causing a problem and you want to deal with it - and you want the language out there so that you can deal with it.

In Amazon's case, their bandwidth pricing isn't really defendable. It's just crap. However, sometimes you're trying to offer something reasonable, but need to make sure that a customer doesn't end up abusing something. For example, Chia is a cryptocurrency that will basically wear through SSDs (it's a proof-of-space system). There aren't explicit limits on how frequently you can write to a disk from most hosting providers, but Chia goes beyond what normal usage would do to a disk. Chia farmers would rather burn someone else's SSD that they're renting than their own. But no one at most hosting providers was probably looking at how frequently people were writing before noticing "hey, why are the disks failing faster than we'd expect?"

They probably haven't taken action on it because they probably haven't noticed it being a problem. But if you're a whale of a customer and suddenly your data transfer charges drop off a cliff, someone might end up looking into that and seeing what's going on.

Re: Slashing data transfer costs in AWS

#176

Post by danielklnstein »

kevincox wrote: post #162
danielklnstein wrote: post #20

Earlier quoted context omitted.

I'm not sure how this could be removed - the fundamentals behind it are basic building blocks of S3. Maybe raising the cost of transient storage? e.g. If you have to pay for a minimum of a day's storage - but even if that was the case this would still be cost-effective, and at any rate it seems very unnatural for AWS to charge on such granularity. + I would guess that S3 is orders of magnitude more profitable for AWS…

It would be fairly easy to change the pricing policy. GCP did something similar for cross-region https://cloud.google.com/storage/pricing-announce#network . This is pretty severe because it seems to affect all reads. However I can imagine an alternate implementation where the source AZ is tracked when data is written and egress fees are charged when the data is read (as if the data was always stored in the source AZ)…

Yeah, I see what you mean - that'd indeed render this method ineffective. Like you said I'm sure this would bother a lot of customers, but it's not a completely unrealistic overhaul of S3 pricing.

That being said, that'd be sort of "mean" of AWS to do - the data is already replicated across AZs whether you pay for it or not because of how S3 works.

Re: Slashing data transfer costs in AWS

#177

Post by icehawk »

overstay8930 wrote: post #143
echelon wrote: post #105

Earlier quoted context omitted.

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

"this is what we're paid for" Nope, it's what YOU'RE paid for. I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed). I have rarely m…

Anyone serious wouldn't "drive to the colo to swap out a failing line card" they keep have excess capacity and spares in the colo, and have the on-site personnel from the facility replace it.

Honestly just sounds like the environment you describe has greater organizational issues not related to on prem vs cloud.

Re: Slashing data transfer costs in AWS

#178

Post by snug »

overstay8930 wrote: post #91
andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…

AWS and GCP are giving companies like Apple huge discounts so someone could say something like, "Even Apple iCloud is in AWS and GCP because of how economical it is"

There is too much nuance to say one is better than the other. In some cases using a IaaS is more economical, in other cases it's not.

For Apple, the same is also true[0] to say "Even Apple is running their own datacenters because of how economical it is"

0 - https://dgtlinfra.com/apple-data-center-locations/#:~:text=a....

Re: Slashing data transfer costs in AWS

#180

Post by rrrix1 »

overstay8930 wrote: post #152
chongli wrote: post #138

Earlier quoted context omitted.

Both on prem and cloud require people familiar with them and cloud-engineers are in no way cheaper. I think the real story is a bit sordid: office politics. On-prem and cloud are different skillsets. Companies that have been around for a while can end up with both on-prem and cloud experts who end up competing with each other, often on separate teams. Throw in some slick consultants from Amazon who are able to bend t…

Cloud engineers can do the job of 4-5 on-prem people. Our AWS devs don't need to be BGP or ZFS experts, they just need to be AWS experts.

Hilariously ironic, with a sufficiently large cloud footprint, things like BGP (and more/other internetworking protocols) and OpenZFS become required skillsets. I have firsthand experience of this. :)
Post reply on HN