Live data from Hacker News

AWS bill capping feature request thread still unanswered after 10 years

forums.aws.amazon.com

141–150 of 170 posts

Re: AWS bill capping feature request thread still unanswered after 10 years

#141
Disclaimer: Work for AWS Support. Good and bad things about my employer. Opinions my own.

One of the biggest questions here is what are customers asking for? And the necessary follow-up question: Do customers want to actually bear responsibility for their choices?

Recently, I had a customer lodge a support case because their programmatic access keys were leaked. A malicious actor was then able to use those credentials to exfiltrate their S3 data, and delete it. Now the S3 data is being ransomed.

The customer opened a case asking if there was any way to get the data back. If we had to go the 'extra mile', the customer demanded we do that.

The answer is simply: No. We can't get that data back.

Customers demand that they own the data they upload into S3. They don't want AWS to be able to read the data, nor do they want us to store the data internally as a backup. That's what customers demand, and what AWS gives them.

Now something regrettable happened (a customer got pwned) and now a customer wants AWS to bear the responsibility for backing up the data. I bet a week ago they would've demanded that they have absolute sovereignty over their data. Shit changes when shit hits the fan.

---

Hard caps are doable, but the question is: Do customers want responsibility for this feature?

I read a comment wherein someone's startup was killed due to $30k of bandwidth costs because of misconfigurations that led to users abusing their platform. That sucks. It's not in AWS's interest to put their customers out of business.

But let's look at the flip side. What if someone in the finance department puts a hard cap on an account. But then a tweet goes viral, and business is pouring in. Everyone, in their rush to keep everything scaling appropriately, forgets there's a hard cap. They hit the business and now there's a hard outage in the middle of the biggest business event they've ever had.

Who's the customer going to hold responsible for the outage? As someone who deals with customer misconfigurations on AWS for a living, I assure you the customer won't be calling themselves demanding an explanation.

What happens if the person in charge of budgets doesn't lift the cap before the Christmas holidays? Again, hard down. Again, customer won't be mad at themselves.

---

It's better to let AWS run as the customer configures it to run than bring everything to an abrupt stop. Billing alerts exist, and you can use Lambda to turn off resources that are above their billing threshold.

But doing a hard cap on an account? Something a subsection of customers might want, but something not many customers will want to take responsibility for when something goes sideways.

Re: AWS bill capping feature request thread still unanswered after 10 years

#142
post #83

Earlier quoted context omitted.

There are no billing limits, but there are resource limits set by AWS upfront. I had to create many support cases to raise this or that limit. (for example we had a limit if 100 concurrent lambda workers, then 2.5k if I remember correctly). Some number of active ec2s, some total TB of storage etc. We were hitting those limits pretty frequently despite spending over £50k per month with them (mostly dev and test servic…

I'm pretty sure that AWS' service quotas exist more as a guardrail to prevent customers from accidentally spinning up 1000 instances instead of 10, which would not only leave you with an eye-watering bill, but affect resource availability for other customers. They're usually quite happy to increate the quote if you contact support.

Most limits exist within what AWS has determined is 'normal' usage. Once you pass that, you can request a service limit increase.

Service limit increases are typically only denied when raising the limit would negatively impact the availability of the service (noisy neighbor issues, for example), or if the customer is needing a limit increase because they're trying to use the service in a way it wasn't designed.

Re: AWS bill capping feature request thread still unanswered after 10 years

#143

My understanding is that this is a difficult problem to solve "perfectly" due to lag between incurring a cost and recording the cost. I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud se…

Pay up front. Offer services up to total accumulated paid amount.

Re: AWS bill capping feature request thread still unanswered after 10 years

#144

It hasn't been implemented because it is nonsensical. So you hit your limit. Is all your storage deleted? You can't receive an alert because that costs something (even if it's a fraction of a cent) to send. Are your domains forfeit? Audit logs destroyed? There's no reasonable way to implement this. Billing alerts are the best you can do on this problem and I think it would be a good faith move for AWS to enable some…

What the hell good is a budget alert when you’ve launched 2000 high memory instances because your programmer screwed up?

There's other things you can use for this, such as CloudWatch Alarms.

And where are your code reviews?

Re: AWS bill capping feature request thread still unanswered after 10 years

#145

Earlier quoted context omitted.

Anyone who is personally liable for the the bill should be terrified of AWS. It's enough to lose your business to a usage spike, much worse to lose your home and car. I struggle to consider a scenario where setting up a limited liability organisation wouldn't make sense, even if only for interacting with Amazon.

Is there any record of Amazon going after people (and not companies) for unpaid bills? More often than not, I hear stories about how AWS support zeroed out the bill instead.

https://news.ycombinator.com/item?id=22719573 and the rest of that thread just makes me clench up.

As sibling reply also says, even if they do eventually zero it out, do you want to owe the richest guy on the planet $100k? He has law firms like you have pairs of socks.

Obviously everything I'm saying goes for unlimited-billing cloud infrastructure, where your customers (or your mistakes) generate cost out of view.

Re: AWS bill capping feature request thread still unanswered after 10 years

#146

Earlier quoted context omitted.

Disclosure: AWS employee. Support specifically. There are good things about my employer. There are bad things about my employer. AWS does indeed obsess about the customer. Every step along the chain there is someone there advocating for the customer. There are mechanisms to keep the customer in mind even for the developers who actually code the service and don't talk to customers on a daily basis. I've had many, many…

$0 ingress charges and >> $0 egress charges make AWS roach motel.

So basically you're complaining about pricing? Unlike other cloud providers, AWS has never had a price increase for a service. Just decreases.

I guess 'customer obsession' would be giving away everything for free?

Re: AWS bill capping feature request thread still unanswered after 10 years

#147
post #25

Earlier quoted context omitted.

That is an argument for why these overages occur. It isn't an argument for why customers should eat that cost rather than Amazon. In fact Amazon is in a much better position than customers to absorb those costs. Sure, they'd have to increase rates slightly to cover it. But it would give customers peace of mind. And if the costs become exorbitant, Amazon is in a better position to improve their own systems to reduce t…

I would argue that AWS does not have a "Customer Obsession", and that's exactly why it's Amazon's most profitable business (by far) and the underwriter of all of Bezos's ambitions.

I am an AWS employee as well, and I'm definitely one of the biggest critics of the company. That being said, AWS is definitely still customer obsessed.

I feel dirty defending AWS, but this is one case I'd give them the benefit of the doubt. There must be _a_ reason they haven't implemented this yet and that reason must be somehow protecting the customer. "Customers want this" ends the discussion around here. You must have a really good reason to disagree.

Re: AWS bill capping feature request thread still unanswered after 10 years

#148

My understanding is that this is a difficult problem to solve "perfectly" due to lag between incurring a cost and recording the cost. I believe GCP currently has the best feature for this. You can set both billing alerts as well as caps. However, I also believe that it can take up to 24 hours between incurring a cost and it showing up on your billing report. So even on GCP (which is the most forward-thinking cloud se…

As I posted last time this came up here:

This is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount. If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get a bill for $1000.

With an actual hard limit you create a financial incentive for the provider to minimize this overrun. Yes it might be difficult to fix but I assure you, if hard limits existed, the technical issues would be solved soon enough because now there's a reason to invest in a solution.

Re: AWS bill capping feature request thread still unanswered after 10 years

#150

Earlier quoted context omitted.

> and I can't really see any reasons it couldn't work. You’re assuming the existence of a function which can extrapolate the daily/monthly/etc costs. If I upload a 10GB file at $0.10/gbmo, how much will it cost me? We have no idea because we don’t know how long it will be up there. We have a process that downloads and streams massive reports into S3, then a process is immediately kicked off to handle processing and i…

You're over complicating things, I'm not assuming the existence of any magical function (other than the one Amazon already uses to calculate your bill). All I want is to be able to say "I do not, under any circumstances want to spend more than $X per month on this service." Amazon just needs to keep track of usage, and start rate limiting/returning 402s/blocking writes when the monthly limit is hit. If I estimate I'm…

> Amazon just needs to keep track of usage, and start rate limiting/returning 402s/blocking writes when the monthly limit is hit.

You're accruing costs even if you stop reading and writing. You're paying for the on-going storage, running of instances, etc.

What you're describing does not implement what you're describing it as.

E.g.,

It's the first day of the billing cycle, so our bill is effectively $0. So no limits in place on writing/rates/etc. We upload 100TB of data. No rate limiting or blocking because our current bill is under the $800 limit.

By about the 8th of the month, we've now hit the $800 limit you set. If you don't go hit the "let me exceed my billing limit" then you want Amazon to "stop the service".

You want:

  1. Under no circumstances to spend more than $800/mo.
  2. Your data to be retained.
This is not possible. There are two paths forward here:

  * Amazon retains your data: You need to continue paying for storage. Your bill at the end of the month will be ~$2200. Fails #1.
  * Amazon "stops the service" (storing your data) and under no circumstances exceeds $800/mo in charges: They delete your data. Fails #2.
You're asking for Amazon to provide you services and just not bill you for them if you don't want to pay for them. This isn't going to happen.

Alternatively, if you want them to not accept a write if it _would_ have led to you being over your budget, see my original comment about predicting the future.

Post reply on HN