Live data from Hacker News

Accident Forgiveness

fly.io

281–290 of 310 posts

Re: Accident Forgiveness

#281
post #43

Earlier quoted context omitted.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than…

We’ve now gone from hard billing caps to “soft billing with alerting”.

If you look at lots of these threads you’ll see that many people don’t want to provide phone numbers, lots of people ignore emails, even repeated ones, directly to them, from billing.

This isn’t a technical problem, it’s a service problem. I can see the hn posts already “my site went viral and shut me down”

Re: Accident Forgiveness

#282
post #56
post #44

Earlier quoted context omitted.

That would be what you would pay in the absence of competition. This is an incredibly competitive space.

Then how do AWS, Azure and GCP charge 100 times the amount for bandwidth as other hosting providers and the IP transit quote sitting in my email inbox?

IMHO, there's a couple parts to this.

Part 1 is bandwidth prices vary tremendously by location, but clouds would like to have customers in expensive regions too, and if US overcharges at 100x, and Brazil overcharges at 25x, the customer will only have to pay 2x for bandwidth in Brazil. Not a lot of cheap hosting in Brazil, from what I've seen, but there are a lot of users in Brazil, attracting cloud customers justifies putting more cloud hardware there which benefits the cloud.

The other part is that bandwidth is easy to measure and broadly correlates with general usage. On a shared system, you can't really meter watt hours, but you can meter bandwidth. Bandwidth charges are how the difficult accounting is reconciled so that the cloud can charge enough to hit their desired margins. If bandwidth was just a cost plus, other things would cost more; if everything had to be cost plus, accounting would be much more difficult for everyone (and it's already pretty non transparent)

Re: Accident Forgiveness

#283
post #121
post #43

Earlier quoted context omitted.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

My preferred option would be to have an optional billing cap that I can enable knowing full well that if it is exceeded the service would be terminated (obviously with notifications as that cap is approached). I could then apply this to simple hobby projects and such, while not having the risk of termination apply to more serious applications (though a 'soft cap' would be nice here so that I could still receive notif…

aws has billing alerts that trigger lambdas.

> aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | xargs -n1 aws ec2 stop-instances --instance-ids

Will stop all ec2 instances.

The real fix is scoping credentials on aws - if you don’t use an account or role with limited permissions then even if they had this toggle the first step in an attack would be to disable this option.

Re: Accident Forgiveness

#284
post #275
post #48

Earlier quoted context omitted.

Competition only matters for new contracts. Once you have picked a provider and made a big enough infrastructure investment, there's no realistic path to switch to someone else.

It really depends on what your application is and how much critical functionality relies on hard to replace proprietary services. If you store a lot of data, transfering it is a big deal too, of course. But really, computers are computers. It's easy to move software between them. It just takes motivation, effort, and time.

> It just takes motivation, effort, and time

In business terms, you just said "cost, cost and cost".

Re: Accident Forgiveness

#285
This thread is full of people complaining that they just want the ability to set a hard billing cap. And yet, providers continue to insist that no serious customer wants this and/or it's impossible to implement.

What would it take for providers to listen to real customers here?

I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If you want my $25k, I need to ensure that I don't spend more than this amount.

As is, my solution is to use old-school pre-rented, long-term contracted commitment VM hosting providers. This is really the only way to guarantee that you are paying an exact amount and no more.

But, I really would like to use a more scalable system that didn't require pre-provisioning. And, I wish people would believe the customer when they say something and not continue to gaslight us.

Providers say it's impossible, but I don't see how it would be so hard. Here's my sketch of how it could work:

The main component is a system that monitors billing events and watches for the slope of the bill to ensure that there is enough runway to stay under the cap. Optionally, they could implement rate limits on resource creation to ensure that a sudden surge doesn't outrun the monitoring.

You also need notifications for when the projected spend exceeds the cap. Optionally, you could implement a soft cap, where no new resources can be created.

And finally, you need the hard cap where things start to get deleted. If they're feeling generous, the provider could implement a period where VM's/lambda's/etc are shut off, blob storage is not accessible, and so on so that the account holder has some time to fund the account and/or fix whatever is causing the overage.

That set of features are all totally within the competency of a cloud provider. Knowing how much things cost, billing for them, and turning them on and off is their main business. And I can't believe that they expect us to believe that it's impossible to do that tracking.

This is how I do system migrations. There are escalating warnings until one day the service is shutoff for an hour and turned back on. That wakes up any laggards that missed the dozen or so communications over a three-to-six month period. Finally, after a few more days, the service is shutdown for good and then data is deleted a week after that. Though I almost always keep a copy in cold storage. But that isn't necessary as a provider with a limited relationship to the client.

Re: Accident Forgiveness

#286

This thread is full of people complaining that they just want the ability to set a hard billing cap. And yet, providers continue to insist that no serious customer wants this and/or it's impossible to implement. What would it take for providers to listen to real customers here? I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If…

Nobody's gaslighting you. It's not impossible to build this, though it is much more difficult than it seems (cloud billing is a large-scale eventually-consistent distributed system, and if you've done any distributed systems work the issue with plugging a system like that directly into a control loop should be obvious). It's just expensive to build, and disproportionately serves the interests of customers who aren't running real apps.

At $25k/mo spend, you can talk to many cloud providers (certainly including us) to work out an "I can't get invoiced for more than $25k" solution which will not involve having your app turned off abruptly when the 2,500,001st cent get spent in October. What you'll notice in this thread is that people generally want billing caps for accounts they plan to spend, like, $10/mo on. And we can build cap systems for those people --- but they'll involve turning parts of the platform off for them.

I'm really pleading with people with strong opinions about hard caps to do the exercise of working out how these billing systems work. There are a huge number of apps running here, across a huge fleet of physicals, running in almost 40 regions around the world. Each of those apps has several different kinds of resources that meter at different granularities and incur different costs. Speaking as a witness to the creation of a new billing system just a month or two ago: it is kind of a miracle that these things work at all. Do the thought experiment, read some Call Me Maybe posts, and then tell me it's obvious that this feature should be straightforward to build.

Re: Accident Forgiveness

#288
post #43

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.

I understand it’s super hard, but Azure has pay as you go and credits that do exactly this. Seems like if you were designing a billing system it’d be a good idea to include this feature.

Re: Accident Forgiveness

#289
post #16

> and may only come within tens of dollars of accurately predicting your water bill. Whenever I've had to water new grass seed I've always been surprised by my water bill.

my well only costs me electricity and has no meter - well, i have a meter somewhere, i just never installed it. I know for a fact it uses 9A to run, so it would cost me 30¢ or so, all-in, to run it for an hour. it's somewhere around 10-12GPM, so 600-700 gallons for 30c I've wanted to put it on solar the entire time i've had it but the start current is 18A and an inverter that can handle that is (or maybe was, idk) re…

Inverters have gotten a lot more capable and quite cheap; you should take a second look. Also look up "soft starters", they're devices designed to reduce start current on motors. They get mostly used for solar and generator A/C, but would work with a well pump, and might put it in the range of your current inverter.

Speaking as someone who has had to run a gasoline generator to pump well water and is about to move it to solar, with great rejoicing.

Re: Accident Forgiveness

#290
post #97

Earlier quoted context omitted.

There’s a guy on Reddit a couple days ago showing off his softball sized watermelon he grew with a $100 watering bill. The struggle is real.

$100 is like 14HCF where I live (though it includes sweage). [edit] 14, not 12HCF.

That’s the problem with irrigation. You get charged for the sewage cost of that much water even though it doesn’t end up there. Unless you’re a farmer.
Post reply on HN