Live data from Hacker News

Accident Forgiveness

fly.io

241–250 of 310 posts

Re: Accident Forgiveness

#241
post #59

Earlier quoted context omitted.

This made me think. There is usually some "hard" cost limit X that you cannot / don't want to afford, so terminating the service is preferable. There is also usually some "soft" limit Y Y than face an outage. But a hard limit would have to be set to X to avoid that outage, and if it gets exceeded, you'll face a bill of X and an outage. So what a customer would actually need is to specify both X and Y, with the rule:…

It would, of course, be just mean and unscrupulous for a cloud vendor to look at the number you have set as being ‘the absolute most I am willing to pay for this service’ and then optimize their pricing offer to you specifically to make sure they go right up to that line and no further.

I didn't mean to imply that. A hard limit at X that stays inactive X, leaves you with a bill of X and an outage is the easiest approach from a technical side: Terminate the service when X is reached, and bill exactly what was provided. It is something you would instantly come up with when asked to implement a cost limit, and you don't for a second put yourself in the customer's position.

Of course cloud vendors do put themselves in the customer's position, and that's why they say that customers would not be happy with a limit, even though they are asking for it.

Re: Accident Forgiveness

#242
post #59

Earlier quoted context omitted.

This made me think. There is usually some "hard" cost limit X that you cannot / don't want to afford, so terminating the service is preferable. There is also usually some "soft" limit Y Y than face an outage. But a hard limit would have to be set to X to avoid that outage, and if it gets exceeded, you'll face a bill of X and an outage. So what a customer would actually need is to specify both X and Y, with the rule:…

Is this a soft limit or a trajectory prediction? I think there isn’t such a thing as a soft limit. Nobody wants to spend any money really right? But you need to spend some to avoid losing service. That’s just a cost you don’t like but need to pay. I definitely get the idea of: I don’t want to spend X so if it looks like I will, terminate service at Y. But I think that’s a special case of the general situation, I want…

> Is this a soft limit or a trajectory prediction?

I don't know. I just tried to frame the problem from a customer's point of view, because cloud vendors' statement that customers would not like a limit is (IMHO) limited by their POV. Customers do want a limit, but not the way that cloud vendors would implement it. I think a huge part of the problem is understanding what exactly it is that you need when you use a cloud service. (This is varying from customer to customer, and from service to service, of course. You usually have important services that must be running, and others where an outage would be unpleasant but not critical.)

> Nobody wants to spend any money really right? But you need to spend some to avoid losing service. That’s just a cost you don’t like but need to pay.

That is not the issue. From a customer's POV, I would be ready to spend extra to keep the service running, but there is a limit where I'd prefer an outage because I can't bear that much. There are two problems with that: First, the limit is blurry. Second, a simple hard limit would leave me with a huge bill AND an outage. I would want to be able to choose one of those evils, not be left with both. And these two problems compound.

> But I don’t know much about this at all. My whole experience was accidentally getting my own personal self a $500 AWS charge and then deciding they cloud services were dumb.

I don't think they are dumber than the alternative. If you run your own hardware, you have a hard limit in both cost and computing power. You could technically get that with the cloud too, but it is not usually offered because it doesn't really solve the problem, but neither does it for for your own hardware.

That said, it would be nice if the major clouds would offer a "hard limit" option, but it really only works for "unimportant" applications that are cost-sensitive and can take an outage.

Re: Accident Forgiveness

#243

Earlier quoted context omitted.

Someone paid those bills to somebody. AWS has costs incurred from it (support, opportunity, etc) and they paid them.

From that point of view, the cap seems more like a common-sense self defense feature that the cloud provider would want to implement. But we have cloud providers in this thread saying they don’t want to implement caps, so, I dunno.

From my technical understanding from a few friends who may or may not know what they are talking about, the caps issue has to do with the processing delay between billing events being emitted and understood. That by the time the billing events have been processed and action taken, the damage would already be done in all but the most extreme cases.

Secondly, the best solution is to simply stop everything. But now the customer has to cold-start their entire infrastructure, which may actually cost more than paying the bill.

Thirdly, it is likely that customers will set a billing limit and then forget about it years later. Suddenly, they've got a complicated infrastructure setup spanning the globe. They finally hit a scale where they hit their billing limit that they had completely forgotten that Bill configured in their early days (who doesn't even work there anymore). Suddenly, the entire global infrastructure is shut down in the middle of the night.

That's the gist of what my friends said.

Re: Accident Forgiveness

#244

Earlier quoted context omitted.

Metering, pricing, and billing is way more complicated than you might assume. There are historical posts here with more details if you search. In short its all async, theres variable lag, theres multiple “types” or dimensions to metering, the prices vary by SKU + customer + previous metering or billing value + other SKU usage, and billing is not uniform across customers. Imagine needing to accumulate all the metering…

No, this is motivated reasoning brainrot. It's overcomplicating the problem by hyperfocusing on a specific implementation that's already been judged infeasible to justify not doing it. The actual problem that people want solved is "the customer wants predictable, budgetable upper bound periodic cost". You are not unique in offering a service where this is a desirable property. Realigning this sort of cost structure i…

I dont understand what youre on about. The post I replied to, and many others, use “caps” to refer to limits of service based on billing. This is an endless source of comments on every “cloud” topic. I provided a very brief overview of why large billing systems are more complicated than expected and have an impedance mismatch to the stated desire. But sure, tautological brain rot. Got it. Im sure you have a wealth of experience with metering, billing, and pricing for billion dollar revenue streams.

Now if you have some _other_ proposal for how billing and service limits could function Im legitimately interested. But I dont see anything at all specific or actionable in your replies. Insurance is interesting for _some_ facets. Im curious how you think that aligns with dynamic resource utilization and what happens at the boundary.

Re: Accident Forgiveness

#245
post #43

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

Here's the thing. We have apis for everything and their grandmother. You create an instance and there are apis for adding tags, labels, nicknames...but not for spending caps? I understand I don't know all of the complexities involved, but if you can bill by the second or by the hour, you can certainly alert by the same metrics.

We have been measuring CPU, MEM with extreme granularity, how about considering price as a resource and measuring the same way, so that a service with a price cap can self manage and self terminate according to some priority field?

This might not be the actual solution, but we have been at this for a very long time, seems like there is not even a hint of an attempt at solving it by the giants. This is about incentives, sorry.

Re: Accident Forgiveness

#246

Earlier quoted context omitted.

I don't think you've thought very carefully about this, because there are real technical problems here. If we did a cap feature, enabling it would involve disabling parts of our platform. We're just going to go back and forth on this, because I'm not going to write you an essay on this, and you're going to keep saying "it's a business decision" and I'm going to keep knowing you're wrong about that. For the nth time o…

> If we did a cap feature, enabling it would involve disabling parts of our platform. What I'm saying is that capping billing is not the same thing as shutting down parts of the platform. You're redirecting complaints about the latter by focusing on the difficulty of doing the latter in real time, which is granted but gross overkill. The former is something that is not only doable, but something that already happens…

Ok, so you think unintended usage should be costed out by the provider. Sure, T&S and support definitely have a handle on the topic. Now what? Because _today_ it’s already baked in to the P&L and pricing. You want the provider to give it to you as a line item that you dont control? Or to do actuarial evaluation of your footgun propensity to charge you more or less? Why? Im totally missing what solution your suggesting, the problem it solves, and why the provider _and revenue generating customers_ care.

Re: Accident Forgiveness

#247

Earlier quoted context omitted.

I don't believe you saw me say that.

> If you're really only looking to spend $50, we should put our cards on table and say that we're generally not making product pricing decisions with you in mind. i think is what gave that vibe off. I was on a read-only phone in bed and saw the quoted message. got up and logged into the PC to think about what to say. It may be time for cloud providers to dissuade small users away, instead.

Hes telling you the reality. Cloud providers dont want to _dissuade_ hobbyist users. But thats not their target market and they will make business decisions based on the revenue generating customer profile. Thats just being frank and honest.

Major cloud-style companies dont drive significant revenue from your $50/mo cohort. And a $5/mo dev account is basically courtesy for the sales pipeline. The vast vast majority of revenue is “enterprise” sales with private pricing and spend in the hundreds of thousands to millions range.

Re: Accident Forgiveness

#248
post #153

Earlier quoted context omitted.

You are comfortable scaling a service down to zero when configured to do so though. Of course that comes up again when anyone sends a request, but that feels sort of in the same category. That said, I do understand that building your service for people like me that’d rather be restricted to just $50/month doesn’t really make sense.

I don't think anyone with a serious app running on us will use a cap. Just stay fixated on this scenario: a deploy-only token gets stolen, and the attacker (like most cloud attackers) uses it to stand up a bunch of Monero miners. As a consequence... their main app goes down? Who would be OK with that?

[deleted]

Re: Accident Forgiveness

#249
Isn't it easier to set hard limits on the account and notify the client when they're close to them so they can decide to increase them on their own if needed?

But then you run into the risk that the client will never increase those hard limits and ... pay less than they could. Not good for business.

Re: Accident Forgiveness

#250
post #43

Earlier quoted context omitted.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

Here's the thing. We have apis for everything and their grandmother. You create an instance and there are apis for adding tags, labels, nicknames...but not for spending caps? I understand I don't know all of the complexities involved, but if you can bill by the second or by the hour, you can certainly alert by the same metrics. We have been measuring CPU, MEM with extreme granularity, how about considering price as a…

I was pleasantly surprised when I was messing around with the Google maps API and found I was able to adjust a quota to put an upper cap on daily spend.

It made me feel much more comfortable hacking around and not needing to worry that I'd accidentally create a render loop or something that could rack up a bill whilst I wasn't looking

Post reply on HN