Earlier quoted context omitted.
>It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. Making the worst case scenario no worse than traditional infrastructure.
Correct. That argument assumes that every penny spent autoscaling has a positive ROI.
Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
361–370 of 397 posts
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#362The main reason why cloud providers can't provide this killswitch feature everybody talk about here is probably that they just can't. I don't think any of them is capable of billing in real time. So the monitoring would have to happen on resource consumption. This is already enforced with resource limits in many cases.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#363Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#364Earlier quoted context omitted.
You're probably correct. It really makes you think twice about setting up some of those cloud services without a hard limit cutoff.
I normally receive only my "part" of my corporate credit card statement, but earlier this year I was sent more of it. That's when I found out the card has a credit limit of over €50,000. Reading this, I wonder if we should contact the bank and ask for another card with a lower limit, to use with various cloud services. We are 99% on-premises, but have about €200/month in various GCS/AWS usage.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#365Cloud solutions are immensely and, for most cases, unnecessary complicated. In addition to that, they have the ability to kill any dream of yours with invoice far exceeding your darkest forecasts in less than an hour.
The time OP spent solving this issue could be better spent learning about self-hosting and avoiding this trouble all together.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#366Earlier quoted context omitted.
Different expertise. I'm guessing you are neither?
Incorrect guess, and it still doesn't really change anything. You're just playing with words. It's no more useful than a full thread arguing about a misspelling; Just pure noise. Software engineering could learn a lot from, say, civil engineering. It could also learn a lot from interface design and I'm sure even microbiologists and astronauts could teach us a lot. Engineering is not special.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#367As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…
If I were in this situation I would probably offer 10% of the bill to the employee as a reward for their help.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#368Earlier quoted context omitted.
> And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. What? Why wouldn't this just be an opt in thing? It could even be tied to the account being used. It's not like AWS accounts are expensive or hard to setup. If a user opts in to the "kill if bill goes too high" and they kill a critical portion of their business, then t…
> What? Why wouldn't this just be an opt in thing? "Oh cool, I'll set a $1k cap, never gonna spend that on this little side proj." Fast forward a year, the side proj has turned in to a critical piece of the business but the person who set it up has left and no one remembered to turn of the spending cap. Busy christmas shopping period comes along, AWS shuts down the whole account because they go over the spending cap,…
"Oh cool, I'll only scale to 1 server, never gonna see high load for this little side proj."
"Oh cool, I'll deploy only to US West 1, outages are never going to matter for this little side proj."
There are a million ways to be out of money as a company. Why should this be any different? Why is the singular particular instance one where it is simply intolerable to accept that users can screw things up?
There are lots of things that are "shit experiences" that are consumers fault.
There is an easy solution. Give consumers the option and let them deal with the consequences. There are enough valid reasons to want hard caps on spending that it's crazy to not make it available because "Someone MIGHT accidentally set the limit too low which will cause them an outage in production that MIGHT mean they lose money"
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#369Earlier quoted context omitted.
It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…
You've explained why it's hard for Google to not give me resources I can't pay for, but that's not what I care about, or what I'm asking for. What I'm asking for is a feature where I set a hard limit of $100 and that's the most I get billed - if my account accidently uses $5000 of resources before Google reconciles the usage with my budget then Google automatically waives the additional $4900 and then limits my accou…
How will Google automatically differentiate between an "honest mistake" and someone taking advantage of this feature?
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#370Earlier quoted context omitted.
If I'm the only developer on a project and I really need to get to market I might do just that. I sometimes do day hikes on weeknights so this is actually a likely scenario for me.
Do you go hiking alone without your phone? That seems dangerous. And why would you start a test if you won't be there to see the results of the test? Seems more sensible to either leave after you've run the test or wait to do so until you get back.