Accident Forgiveness
fly.io
Accident Forgiveness
1–10 of 310 posts
Re: Accident Forgiveness
#2This shouldn't be a selling point, it should just be table stakes.
Re: Accident Forgiveness
#3>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does.
If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs.
Good on Fly.io for trying to handle such situations more sensibly.
Re: Accident Forgiveness
#4>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for so…
There's plenty of situations you don't hear about.
A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
Re: Accident Forgiveness
#5To be clear, in the case of those abusing the policy:
> We reserve the right to cut you off.
Re: Accident Forgiveness
#6>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for so…
> If the incidents that made the rounds here in the last few months are any indication
They really aren't. There's a huge world out there beyond the HN front page.
Re: Accident Forgiveness
#7>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for so…
There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
Re: Accident Forgiveness
#8It's unfortunate that the solution to cloud pricing complexity that all providers are adopting is – add even more complexity on top.
The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be smoothened? None of this can be deterministically calculated from the pricing page. And there's no way for you to do these checks yourself before deployment because you have no idea what this logic even is. So you are entirely at the mercy of the provider for "forgiveness".
Who wants to bet that some provider is going to launch "AI cloud billing" within the next year?
Re: Accident Forgiveness
#9This shouldn't be a selling point, it should just be table stakes.
You can say this for a whole bunch of new features people role out but this doesn't erase the fact that they are very much not the stakes and introducing them is great because it raises them!
Re: Accident Forgiveness
#10Earlier quoted context omitted.
There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.