Live data from Hacker News

Accident Forgiveness

fly.io

261–270 of 310 posts

Re: Accident Forgiveness

#261
> The Fully Automated Accident-Forgiving Billing System of the Future (which we are in fact building and may even one day ship) will give you a line-item veto on your invoice.

For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over budget than ever see a bill that's higher than our net worth. A line item veto somewhat alleviates that concern but not fully.

They indicate at the end that they're going to do something along these lines, but what they're describing there also seems over-engineered compared to a simple circuit breaker that kills the account or some subset of it. Is there a good reason for these providers to avoid implementing that feature, which on a naive look seems far simpler than what they've actually proposed?

Re: Accident Forgiveness

#262
post #50

Earlier quoted context omitted.

Having flashbacks to the time where we had paid for a server and were paying for rack space for a customer and they were refusing to pay their bill. Our lawyers told us in no uncertain terms that turning off the server would be a terrible idea. “Obstruction of service” is the term that comes to mind.

Cloud squatting?

Node eviction

Re: Accident Forgiveness

#263

Earlier quoted context omitted.

No one really cares enough about hobby developers as a customer segment to rebuild the billing infrastructure to make this possible. Scalable billing at huge scale is solved at the cost of latency and being "eventually correct" (unless it has changed). To add a price cap feature, eventually correct isn't enough and then you ask yourself who would actually use it and you have to scroll really far down your list of big…

The problem with not caring about hobby developers is that it means developers won't be as familiar with your cloud environment when time comes to pick one for the next "real" project. I would also expect a price cap feature to be useful for experimental/no-approval-required projects at work. In fact, if I ran a cloud project for work as a small team-internal project, a cost explosion would become an even bigger bure…

[deleted]

Re: Accident Forgiveness

#264
post #191

Earlier quoted context omitted.

No serious operator would choose a provider with your implementation of that feature.

I agree that no serious operator would choose a provider whose spending cap feature was "enable this feature and we kick you off the platform".

People will pay to be abused: c.f. Oracle

Re: Accident Forgiveness

#265

This is why I love Hetzner. It has billing alerts. What was so difficult about it that fly had to rebuild their system? Also, do everything except databases inside kubernetes. Deploy kubernetes across multiple clouds via wireguard. Label your instances properly on each provider. Prefer bare metal instances where available. Migrate your workloads accordingly. Force cloud providers to earn your money. Don’t however hav…

I can't read the tone, but perhaps you haven't seen much technical debt. It's possible to have a fairly good codebase, full of good decisions, yet you still can't "get there from here".

Second paragraph was useful, thanks!

Re: Accident Forgiveness

#266

If I were fly I’d implement spending caps just to filter out unserious operators. My implementation of that feature would be when you turned it on fly would just kick you off. No serious operator wants a provider to turn their system off. So if you want that it’s pretty clear you are hosting a silly system. Which likely costs more to support and drives less margin.

[deleted]

Re: Accident Forgiveness

#267
post #8

It's unfortunate that the solution to cloud pricing complexity that all providers are adopting is – add even more complexity on top. The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be s…

I'd recommend thinking of this (and other accident forgiveness schemes from competitors) as a gesture of goodwill that rarely happens rather than an official part of the billing policy. If you actually look at your contract, no cloud provider is going to contractually obligate themselves to forgive your bill, and you shouldn't be planning or predicting your bill based on it.

Isn't the entire point of the article that Fly.io is making this an official policy?

Re: Accident Forgiveness

#268

Earlier quoted context omitted.

From that point of view, the cap seems more like a common-sense self defense feature that the cloud provider would want to implement. But we have cloud providers in this thread saying they don’t want to implement caps, so, I dunno.

From my technical understanding from a few friends who may or may not know what they are talking about, the caps issue has to do with the processing delay between billing events being emitted and understood. That by the time the billing events have been processed and action taken, the damage would already be done in all but the most extreme cases. Secondly, the best solution is to simply stop everything. But now the…

[deleted]

Re: Accident Forgiveness

#269

Earlier quoted context omitted.

No, this is a much harder problem than you think it is; it's, like, CAP-theorem problematic. I think you should do the exercise of working through how these billing systems work.

I reiterate that you are not unique in offering a service where the customer's desire for consistent billing and your willingness to provide it are at odds.

Define “consistent billing” please, I don’t even know what that means and I work in this space.

Re: Accident Forgiveness

#270
post #62
post #56

Earlier quoted context omitted.

Then how do AWS, Azure and GCP charge 100 times the amount for bandwidth as other hosting providers and the IP transit quote sitting in my email inbox?

Why isn't one of those providers roflstomping the other 3 (add OCI to the mix) with better pricing?

I've often wondered about this. I'm guessing large customers can all negotiate extremely deep discounts on bandwidth from all three providers. Smaller customers may not be paying that much in bandwidth that it would be decisive. I think also that if say GCP cut its bandwidth charges by 10x they might attract customers they really don't want.
Post reply on HN