Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accident Forgiveness

fly.io

Post reply on HN
Print view View original

271–280 of 310 posts

Re: Accident Forgiveness

#271

Post by tptacek »

withinboredom wrote: post #243
bee_rider wrote: post #81

Earlier quoted context omitted.

From that point of view, the cap seems more like a common-sense self defense feature that the cloud provider would want to implement. But we have cloud providers in this thread saying they don’t want to implement caps, so, I dunno.

From my technical understanding from a few friends who may or may not know what they are talking about, the caps issue has to do with the processing delay between billing events being emitted and understood. That by the time the billing events have been processed and action taken, the damage would already be done in all but the most extreme cases. Secondly, the best solution is to simply stop everything. But now the…

I'm just going to refer people to this comment.

Re: Accident Forgiveness

#272

Post by tptacek »

lolinder wrote: post #261

> The Fully Automated Accident-Forgiving Billing System of the Future (which we are in fact building and may even one day ship) will give you a line-item veto on your invoice. For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over b…

Something like half the comments on this story are a discussion of why or why not cloud providers do or don't provide this simple circuit breaker feature.

Re: Accident Forgiveness

#273

Post by ensignavenger »

zipmapfoldright wrote: post #128
paxys wrote: post #48

Earlier quoted context omitted.

Competition only matters for new contracts. Once you have picked a provider and made a big enough infrastructure investment, there's no realistic path to switch to someone else.

This is absolutely not true. Customers regularly shift spends in the >= 7-digit between cloud providers. Yes, it takes planning, and it takes many engineer-months of work, but it definitely happens. And also factor in (1) the claim that most cloud growth is ahead of us, eg. moving large customers from on-prem to cloud, and (2) it would be terrible policy to try to charge existing customers more than new customers.

>> it would be terrible policy to try to charge existing customers more than new customers.

Companies do this very, very often. This is part of the reason why they have a "call the sales department for special pricing" option. They can give large contracts a nice discount to get them onboard, then slowly (or in some cases, if they really think they have you hooked, not so slowly) ratchet up the price. This is common in both B2c and B2B businesses.

Re: Accident Forgiveness

#274

Post by lolinder »

tptacek wrote: post #272
lolinder wrote: post #261

> The Fully Automated Accident-Forgiving Billing System of the Future (which we are in fact building and may even one day ship) will give you a line-item veto on your invoice. For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over b…

Something like half the comments on this story are a discussion of why or why not cloud providers do or don't provide this simple circuit breaker feature.

I read yours after posting this and it provides exactly zero information:

> I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal.

"Terminating service is a big deal" how? I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend?

No one is asking for automatic circuit breakers applied to all customers indiscriminately, but I'm not seeing any justification in here for why an opt-in circuit breaker is technically or legally challenging to implement.

Re: Accident Forgiveness

#275

Post by toast0 »

paxys wrote: post #48
tptacek wrote: post #44

Earlier quoted context omitted.

That would be what you would pay in the absence of competition. This is an incredibly competitive space.

Competition only matters for new contracts. Once you have picked a provider and made a big enough infrastructure investment, there's no realistic path to switch to someone else.

It really depends on what your application is and how much critical functionality relies on hard to replace proprietary services.

If you store a lot of data, transfering it is a big deal too, of course.

But really, computers are computers. It's easy to move software between them. It just takes motivation, effort, and time.

Re: Accident Forgiveness

#276

Post by sigseg1v »

tptacek wrote: post #43
bitbasher wrote: post #38

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than 10% of my monthly spend if left running for 24h."

This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.

Re: Accident Forgiveness

#277

Post by klabb3 »

bitbasher wrote: post #38

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

Yes, but I imagined that you’d have a ceiling for rolling average / token bucket over a sensible time window, with two limits:

- a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out”

- a hard limit which fails until more tokens trickle back in, without shutting down service

I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing.

Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought

Re: Accident Forgiveness

#278

Post by Dylan16807 »

dlisboa wrote: post #103
belthesar wrote: post #95

Earlier quoted context omitted.

I'll be honest, I think you've got an uphill battle to convince so many of us that are concerned about accidental pricing that this is exactly the thing we actually want, vs. what we feel we want. This position parrots what Vercel's leadership said when someone had a massive surprise bill, and the story made the rounds both here and elsewhere. To be super honest, you might be right too. You might go down a huge engin…

> You might go down a huge engineering effort to build this This is an overlooked issue: billing caps are hard to implement and will likely incur losses for the cloud company that does. Take an object storage service as an example. Imagine Company X has a hard cap at US$ 1000 but some bug makes their software upload millions of files to a bucket and rack up their bill. Since objects are charged in GB/month they will…

If you turn off servers and shut off bandwidth you get rid of the vast majority of expenses.

Storage fees are a lot less risk, but if you want to cap those then you should cap the number of gigabytes directly. That prevents the overage issues you describe.

Re: Accident Forgiveness

#279

Post by taormina »

theusus wrote: post #92
tptacek wrote: post #68

Earlier quoted context omitted.

What do you mean by "default offerings"? Thanks!

https://www.heroku.com/ Checkout their products

Heroku is not Fly.io. Heroku was bought by the MBAs a while back, and they do something different than generic cloud hosting (they are closer to a hosted EBS). If you bother scrolling at all, you'll notice immediately below the fold that the pricing is not remotely as clear as you're pretending it is.

Re: Accident Forgiveness

#280

Post by Dylan16807 »

donavanm wrote: post #244
naniwaduni wrote: post #187

Earlier quoted context omitted.

No, this is motivated reasoning brainrot. It's overcomplicating the problem by hyperfocusing on a specific implementation that's already been judged infeasible to justify not doing it. The actual problem that people want solved is "the customer wants predictable, budgetable upper bound periodic cost". You are not unique in offering a service where this is a desirable property. Realigning this sort of cost structure i…

I dont understand what youre on about. The post I replied to, and many others, use “caps” to refer to limits of service based on billing. This is an endless source of comments on every “cloud” topic. I provided a very brief overview of why large billing systems are more complicated than expected and have an impedance mismatch to the stated desire. But sure, tautological brain rot. Got it. Im sure you have a wealth of…

> if you have some _other_ proposal for how billing and service limits could function Im legitimately interested.

Billing doesn't have to be so complicated you can't calculate it in less than a minute. That's a technical failure. Surely you can imagine a better way? If you really think it can't be better, then it's hard to argue against "brain rot".

Also on most systems it will work perfectly fine to use an estimate of the price per unit when calculating the bill for the last couple hours.

Post reply on HN