Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accident Forgiveness

fly.io

Post reply on HN
Print view View original

301–310 of 310 posts

Re: Accident Forgiveness

#301

Post by veggieroll »

tptacek wrote: post #286
veggieroll wrote: post #285

This thread is full of people complaining that they just want the ability to set a hard billing cap. And yet, providers continue to insist that no serious customer wants this and/or it's impossible to implement. What would it take for providers to listen to real customers here? I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If…

Nobody's gaslighting you. It's not impossible to build this, though it is much more difficult than it seems (cloud billing is a large-scale eventually-consistent distributed system, and if you've done any distributed systems work the issue with plugging a system like that directly into a control loop should be obvious). It's just expensive to build, and disproportionately serves the interests of customers who aren't…

It seems I've committed the cardinal sin of failing to specify my units. So to clarify, it's $25k per year.

But my mistake that aside, I appreciate your reply. I saw in another comment thread that you added that you have a blog post in the pipeline on building the new billing system. So I'm really looking forward to reading that. I've enjoyed reading other billing/payments content. And I'm sure your post will also be insightful and highly detailed.

I think it's so so fascinating that this feature is consistently solved at the contract/legal and customer support "layer" of the stack. That's really unpleasant to me, because it is a lot harder for me to wrap my head around the specifics of how different edge cases will play out.

Like as a programmer, I've built up all these skills on reaching technical documentation, understanding systems, their limits, and complex interactions. But instead of using that muscle memory, I have to try to talk with a human and deal with the seemingly intentional vagueness of the legal system.

It feels to me a lot like the story from Mitchell Hashimoto about dealing with the bank for his startup, where he was dodging calls from his account executive and generally behaving in a way that the bank is not used to from enterprise clients. [0]

I'm ready to admit my behavior is anti-social and irrational here. But, it is what comes natural to me.

This is meandering now. But, I just want to sneak in a bit more info on my use cases, since you also mentioned that people are wearing you down and you just might implement this if forced to.

I build and run internal tools (think CRUD & reporting/analytics) for a small department in an extremely large enterprise. Our stuff is on the order of 99.9% available. So not particularly great but not not terrible. But, others are extremely bad. For example, one vendor has over 36 hours of scheduled downtime per month. And that system is way more critical to the business than mine.

So the standard that my coworkers in the department have come to expect is very low. If the tool is down, they just continue with their day doing some other task.

Many of the systems I manage are also purely background jobs. And no one would even notice if they were down for 12-24 hours.

Lastly, we have external backups for everything (on a different provider) and every system's deployment is automated from creating the VM's, networks, and block storage all the way through to installing system dependencies, the app, and data.

So, if a system were to magically get deleted some day, I'd get paged and have it back up in about an hour. And this is totally fine for our business.

On the other side though, there will be dire consequences for my career if we go over $25k annual spend. Even if the bill arrives and we have to contact support, it will give my management a heart attack and they will absolutely remember come review time.

Given this environment, I'd really appreciate the ability to protect myself against misconfiguration or leaked keys causing me to get possibly fired. The data will be fine. And systems can be restored quickly. But the damage to my reputation, compensation, and future job can't be restored quickly.

[0] https://mitchellh.com/writing/my-startup-banking-story

> Many of the systems I manage are also purely background jobs. And no one would even notice if they were down for 12-24 hours.

Re: Accident Forgiveness

#303

Post by ragall »

ToucanLoucan wrote: post #36
paxys wrote: post #8

It's unfortunate that the solution to cloud pricing complexity that all providers are adopting is – add even more complexity on top. The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be s…

Anyone else remember we had widely available and completely understandable options to rent everything from baseline web-hosting all the way up to private rack services for actual years before AWS came along and apparently all of that was forgotten, Warhammer 40K style? I've rented a VPS from a vendor for going on 20 years now (Holy fuck I'm old) and I've never once been surprised at the bill.

In other words, you have fully reserved compute. That's the way to avoid surprises, but certainly not the way to minimize costs for a large company.

Re: Accident Forgiveness

#304

Post by genewitch »

idlewords wrote: post #300
genewitch wrote: post #293

Earlier quoted context omitted.

thanks, what brand, if any, do you recommend?

I recently did an inverter replacement and my solar guy recommended this model: https://eg4electronics.com/wp-content/uploads/2024/04/EG4-18... Max output is 50A, so it should be plenty for your pump and mine. You can also chain them together if you want huge battery capacity.

thank you, very much.

!! oh wow, they actually support the full open-circuit voltage that my panels can be series linked* to reach! the only thing i saw many years ago was like... the power wall!

Not only that, it can run every panel i currently own all by itself. that is a leap forward, compared to the last time i checked (a decade ago or so, it was depressing to want a separate solar grid, you need the grid tie stuff so you can run your house if the power goes out, but i mostly wanted the solar to manage water and lights and if possible, the small window/wall aircons that i use to keep servers cool.

Re: Accident Forgiveness

#305

Post by turok »

morningsam wrote: post #3

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for so…

We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.)

So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.

Re: Accident Forgiveness

#306

Post by gwbas1c »

Aeolun wrote: post #231
tptacek wrote: post #172

Earlier quoted context omitted.

I don't think anyone with a serious app running on us will use a cap. Just stay fixated on this scenario: a deploy-only token gets stolen, and the attacker (like most cloud attackers) uses it to stand up a bunch of Monero miners. As a consequence... their main app goes down? Who would be OK with that?

I think the cap (if they had any) would probably reflect the amount they stand to lose if their app goes down. If your app brings in $1000/h, you don’t necessarily care about spending that amount on servers. When your costs rise to $10k/hour, you might want to go with the nuclear option. Of course it’s nicer if you can be certain that your provider is going to refund you the excess, but I feel like it’s hard to count…

> Just stay fixated on this scenario: ... the attacker ... uses it to stand up a bunch of Monero miners.

This sounds more and more like an insurance policy; as opposed to a "sudden spike of load because of popularity" situation.

IE, all our talk about usage caps is really missing the point about what needs to be protected against.

Re: Accident Forgiveness

#307

Post by gwbas1c »

donavanm wrote: post #165
gwbas1c wrote: post #51

Earlier quoted context omitted.

One thing I'm really curious about is why caps are so hard? (Perhaps this would result in a more technical blog post?) IE, you clearly don't want to terminate or shut down an account if they get too close to a cap. But what about things like a warning email, service slowdown, ect? Likewise, the old "slashdotted" or "hug of death" might be an appropriate result when something goes beyond a reasonable safety buffer? An…

Metering, pricing, and billing is way more complicated than you might assume. There are historical posts here with more details if you search. In short its all async, theres variable lag, theres multiple “types” or dimensions to metering, the prices vary by SKU + customer + previous metering or billing value + other SKU usage, and billing is not uniform across customers. Imagine needing to accumulate all the metering…

I understand all that; and that's not really the kind of information I'm looking for. (I know deeply that metadata is often more expensive and complicated to process than the data your customer cares about.)

This response explains the problem best: https://news.ycombinator.com/item?id=41334596

> I don't think anyone with a serious app running on us will use a cap. Just stay fixated on this scenario: a deploy-only token gets stolen, and the attacker (like most cloud attackers) uses it to stand up a bunch of Monero miners. As a consequence... their main app goes down? Who would be OK with that?

Reading between the lines: If a customer's utilization suddenly spikes, the assumption is that the customer's revenue will follow. IE, if my utilization goes up 10x, my revenue will go up 10x, so I'll happily pay the bill.

What they are providing is more like an insurance policy against hacking.

And that is the answer that I was looking for.

Re: Accident Forgiveness

#308

Post by mwarkentin »

turok wrote: post #305
morningsam wrote: post #3

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for so…

We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose b…

FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...

Re: Accident Forgiveness

#309

Post by turok »

mwarkentin wrote: post #308
turok wrote: post #305

Earlier quoted context omitted.

We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose b…

FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...

Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.

Re: Accident Forgiveness

#310

Post by kikoreis »

lolinder wrote: post #274
tptacek wrote: post #272

Earlier quoted context omitted.

Something like half the comments on this story are a discussion of why or why not cloud providers do or don't provide this simple circuit breaker feature.

I read yours after posting this and it provides exactly zero information: > I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. "Terminating service is a big deal" how? I can explicitly cancel a subscription after a certain date—wha…

> I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend?

As someone also involved in billing systems for public clouds: in theory there's no difference, but in practice there is a world of difference. This is the sort of situation where the end user is commonly surprised with the consequences of their own decisions. At MGC we have some "soft shut down" processes, and we constantly hear stuff like "I know I said shut down, but this is the one situation where that really didn't make sense"; where examples are "storage which keeps backups became unavailable", "a very simple but critical user auth system disappeared", "I had no idea this was still running on my account", or "OMG not in the middle of the weekend", etc. You can build heuristics and tracking into the system to minimize these situations, but that's a lot of work.

So yeah, it is a valid use case and something many CSPs would like to provide, but implementing something that is actually better than nothing is non-trivial.

Post reply on HN