AWS Customers Rack Up Hefty Bills for Moving Data • Hacker News Board

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#141

Post by mostlystatic » Mon, Oct 21, 2019, 9:09 PM UTC

I recently started running more Google Cloud VMs in the UK instead of the US. It was costing me £2 a day to run the VMs, but somehow I also paid £12 a day for "GCP Storage egress between NA and EU”.

Turns out that by default Google's Docker container registry only stores the Docker images in the US. So each time I launched a VM the Docker image was downloaded from the US. I wrote more about it here: https://www.mattzeunert.com/2019/10/13/reducing-docker-image...

The billing interface didn't show that the Cloud Storage cost was related to the Docker images. I was investigating my normal Cloud Storage use, but it didn't explain why I was being charged so much. Only after a few days did I get the idea that it might be the Docker images that were causing it.

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#142

Post by NotSammyHagar » Mon, Oct 21, 2019, 9:11 PM UTC

tyho wrote: post #101

tyingq wrote: post #94

Earlier quoted context omitted.

Snowball still charges ~3 cents/GB to get it out of S3 and into Snowball.

Well, there is always https://aws.amazon.com/snowmobile/

no prices. don't underestimate the bandwidth of a fully loaded tractor trailer, 21st century style

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#143

Post by rsync » Mon, Oct 21, 2019, 9:16 PM UTC

nostrebored wrote: post #112

rsync wrote: post #107

Earlier quoted context omitted.

"I think AWS's billing is probably already on the falling side of diminishing marginal returns. That is, it's complex enough that more information would tend to hinder customers from getting the best price. Right now, if I plan to reduce my data charges, I have one variable to tinker with." No, it's two variables - the egress charges you refer to and the actual cost to store the data . We[1] have found that it is, as…

And how many PoPs regional interconnects, highly availabile, high throughout connections, cross continent highly available connections do you have? Do you detect failure across these connections? Do you detect grey failures? Do you have a team of infrastructure engineers to look after this network?

We keep all of those to an absolute minimum and avoid as much complexity in our infrastructure as possible.

Which is to say, each of our five[1] regional POPs have a single connection provided through a dumb switch one hop from he.net[2].

They have no interconnection or dependencies to one another.

No routers, no firewalls, no balancing, no failover. When rsync.net fails, it's a very, very boring failure.

We've had zero network outages in the last 60 months or so.

[1] Fremont, San Diego, Denver, Zurich, Hong Kong

[2] init7 in Zurich ...

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#144

Post by GordonS » Mon, Oct 21, 2019, 9:16 PM UTC

nostrebored wrote: post #113

GordonS wrote: post #100

Earlier quoted context omitted.

> AWS is combining all your networking charges into one lump "outgoing data transfer" fee > So it might be fairer if AWS broke out separate line items for internal, incoming and outgoing data transfer This explanation doesn't cut it for me - most (all?) "traditional" VPS providers don't charge for ingress traffic, and I doubt anyone, ever, has charged for internal traffic. So what exactly is 'all the networking charg…

How many of these VPS providers are actually managing global highly availabile network infrastructure?

I would have presumed that the infrastructure cost for each service was included in the cost for each service.

For egress bandwidth costs, I'd assume it included, well, the egress bandwidth cost.

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#145

Post by zmmmmm » Mon, Oct 21, 2019, 9:18 PM UTC

griffinkelly wrote: post #130

tyingq wrote: post #91

Earlier quoted context omitted.

All of them (AWS, GCP, Azure) are priced outrageously compared to high quality, well peered, bandwidth.

My company was getting massive bills on AWS S3 egress. It was one of the reasons we moved to Wasabi for bucket storage; we then had to deal with a huge one time hit for egress, but in the long run the short term cost was worth it.

Do you think Wasabi is sustainable? I looked at them but find it hard to believe they can sustain free unlimited egress forever.

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#146

Post by gwright » Mon, Oct 21, 2019, 9:24 PM UTC

snagglegaggle wrote: post #29

notaround1111 wrote: post #17

Earlier quoted context omitted.

Customers are also to blame, when comparing the costs of two services they tend to look at the cost of an instance hour, or lambda execution and often don't look at transfer costs. Even if a cloud provider had competitive transfer costs they likely wouldn't attract any new customers and would have less margin left over to subsidize the main cost customers look at, $ per instance hour. The less attention is paid to tr…

I think this has more to do with collusion than consumer behavior. On average consumers are very rational, even if their rationality is hard to explain. The issue with per-bit pricing is that a fair agreement for network use would probably look like paying a fee that makes up for the amortization of the network equipment. Anything else is an artificially restricted market created in an attempt to extract more value o…

Why do you think that bandwidth costs should only cover the hardware? What about the electricity, rent, payroll, sales, marketing, administrative staff, insurance, accountants, lawyers, etc.

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#147

Post by Havoc » Mon, Oct 21, 2019, 9:29 PM UTC

lokar wrote: post #110

heleninboodler wrote: post #103

Earlier quoted context omitted.

Ok, help me out: "cold-potato"? :D

Vs "hot potato" Hold onto the packet for as long as you can vs hand it off to your peer as quickly as possible. Most networks do "hot", Google does "cold" since their network is almost always better than that of the peer.

Weird terminology. Thanks for explaining though

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#148

Post by wolco » Mon, Oct 21, 2019, 9:31 PM UTC

asdfman123 wrote: post #50

throwaway5752 wrote: post #41

Earlier quoted context omitted.

I was just reading about advisor termination fee terms which were some substantial multiple of all fees incurred over some past number of years. People under pressure make short term optimizations at the expense of long term strategy. There's nothing more to it. AWS reduced compute and IT expenses in the next few fiscal years, so people jumped all in on it. Solve the problem now. Someone will fix it in the future.

Programmers should campaign for hefty termination fees. Sure, you can fire me, but you'd have to pay me one year's salary. Actually, it would probably just make some programmers even more intolerable.

Instead of options ask for severance term next job interview

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#149

Post by chickenpotpie » Mon, Oct 21, 2019, 9:34 PM UTC

throwaway_bad wrote: post #140

chickenpotpie wrote: post #60

I've actually been working on a library to help mitigate cloud storage lock-in. The idea is to treat cloud storage providers like disks are treated in RAID. For example, you have 3 separate cloud providers. Cloud providers 1 and 2 have every other byte of data striped across them. Cloud provider 3 has parity data. To pull a file you can only need 2 of the 3 cloud providers. If you don't like how a cloud storage provi…

> every other byte of data striped across I assume you didn't mean that literally because I can't see how that will ever work out in terms of cpu cost. I think breaking it up into blocks like what RAID4/5/6 would be better but will still impact the performance of reads. The performance of writes is going to be worse. Not because of the parity calculation but because you will be taking the max latency over all the clo…

Yeah, RAID 4/5/6 are planned for the future. The plan is to offer all of them and let developers choose what is the best practice for their application. RAID 0/2/3 are not CPU efficient, but are great for privacy and security. No cloud provider has the full picture and can't spy on your data and if they have a data leak it won't be anything useful. RAID 1 gives great fault tolerance with no extra latency (except on failures) and prevents vendor lock-in.

Re: AWS Customers Rack Up Hefty Bills for Moving Data

#150

Post by js4ever » Mon, Oct 21, 2019, 9:37 PM UTC

Elect2 wrote: post #120

js4ever wrote: post #105

I have recently talked with a friend about a very cheap way to egress from AWS, at $2.5 per TB with the AWS lightsail $5 instance which include 2TB of egress per instance. This friend have extracted 200TB of backups for $500 instead of $17,000!

So where is your friend's data in? s3? ebs? I'm curious what is the cost to transfer data from s3/ec2 to Lightsail.

Mostly in S3 but S3/EC2 to lightsail is free within the same region :)