Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#131

Post by robrtsql » Thu, Dec 10, 2020, 7:08 PM UTC

AngusH wrote: post #124

jasonpeacock wrote: post #102

Earlier quoted context omitted.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

But we've had disk quotas before that mostly worked? If anything it seems an easier problem than processor time. I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that. Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails. I don't think they deleted files, it was just you couldn't write more…

S3 costs money to keep your files in, even if you're not touching them, so just preventing further uploads wouldn't do much to prevent your AWS bill from increasing.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#132

Post by luser007 » Thu, Dec 10, 2020, 7:12 PM UTC

FastQ wrote: post #106

marktolson wrote: post #68

Earlier quoted context omitted.

Go to billing > cost analysis > filter by resource break down. Azure billing analysis is pretty amazing.

Yeah, but it just shows my database cost which is higher than is listed as far as I can tell.

Could it be listed "hourly" and you're charged "daily"? Add in VAT (equal to 25% in some countries) and you match the 30 times higher than expected charge.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#133

Post by asciimike » Thu, Dec 10, 2020, 7:14 PM UTC

sandGorgon wrote: post #112

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

Extremely technically, the only GCP product that had this feature was App Engine Standard v1, but looks like it's deprecated as of the end of 2019 (https://cloud.google.com/appengine/docs/managing-costs#chang...)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#134

Post by walrus01 » Thu, Dec 10, 2020, 7:19 PM UTC

Alex3917 wrote: post #127

dang wrote: post #110

Earlier quoted context omitted.

Please don't be a jerk on HN, especially in response to someone else's misfortune, even if they brought it on themselves. Maybe you don't need to treat these people better (though why not?) but you owe the community better if you're posting here. If you wouldn't mind reviewing the site guidelines and taking the intended spirit to heart, we'd be grateful. Note these ones: " Be kind " and " Please don't sneer " https:/…

> Maybe you don't need to treat these people better (though why not?) IMHO the best argument for 'why not' would be that it's generally unethical to deploy software without first taking the time to read the manual and understand how your dependencies work. In this case the system wasn't live and the costs of this fuckup were solely externalized onto Google, which is fine because it was in large part their fault anywa…

I think cautionary tale are important - but it's also possible, as I likely did above, to come down on people too harshly. Not everything has consequences as severe as a therac-25.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#135

Post by coddle-hark » Thu, Dec 10, 2020, 7:22 PM UTC

MaxBarraclough wrote: post #10

bharatsb wrote: post #4

Part 2: https://blog.tomilkieway.com/72k-2/

> Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.

I can sympathise with some of these stories, like the ones where an overnight DDOS attack racks up a huge unexpected bill, but this one in particular is just a story of gross incompetence and negligence. The guy hacked together some code in a few days and deployed it to a service with unlimited billing without any kind of sanity checks and without even understanding what he was paying for. He’s an ex-Googler, it’s not like he hasn’t heard stories like this before. And the takeaway? “Oops don’t deploy buggy code” and “I shouldn’t have used the default settings”. OK, sure, let me know how that works out for you.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#136

Post by scrollaway » Thu, Dec 10, 2020, 7:25 PM UTC

nojito wrote: post #108

scrollaway wrote: post #70

Earlier quoted context omitted.

The margin is absolutely not the same across all products. > There is a massive secondary consulting market because of AWS's price obscurities. Its. Not. For. You. AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. That consulting market is…

>AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. This is marketing. It's like saying you want to build a house and the quote you got ends up blowing up 100x overnight. Great example is the 100k credit for startups. You can repeat it's no…

The $100K credit (which I've been granted multiple times) is there because if Amazon can get you to invest serious work into their infra, they'll make up for it in the long run. It's not "lock in", it's sales. The only amazon "lock in" really is their bandwidth-out pricing, which is a sleazy tactic for sure but I'm not hesitant to call it out when it's the case.

You can get the $100/$300/$1000 tier if you are in "just checking it out" solo mode. $5k and up requires either connections, partnerships, or a serious application.

Anyway I don't know what your point is, I'm not even sure if you have one. They're not "marketing" their pricing, nor the fact that you are "forced to design systems that use less resources".

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#137

Post by csharptwdec19 » Thu, Dec 10, 2020, 7:25 PM UTC

thanksforthe42 wrote: post #116

akh wrote: post #78

Earlier quoted context omitted.

How do you take pricing into your design considerations? Does it come with experience from using an AWS service in production and understanding how it's priced, combined with the usage numbers the new system might get? I'm trying to learn more about how engineers currently do this.

Calling programmers "Engineers" is a misnomer. I wish programmers had the prestige it deserved for combining Science, tradition, authority, and art. Engineers are not allowed to use tradition, authority or art. They are restricted to being modern day calculators. Nothing is wrong with either.

The shift from 'Developer/Programmer' to engineer has indeed been part of a push away from creativity towards cookie-cutter work.

An interesting analogue would be the Automotive industry; As time progressed, Companies focused more and more on 'engineering' versus art/tradition/etc. But as the industry evolved, "Flashy" vehicles that took risks became moreso either a halo product for a brand, or relegated to Luxury/Boutique.

And, of course, there was the dark side of this shift; A good example from the 70s, the level of 'engineering' driving the design of the vehicle and it's assembly didn't take into consideration the actual line worker; in Ohio the workers wound up getting overworked, burned out, and in some cases actively sabotaged the product, because they were being treated like automated machines.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#138

Post by csharptwdec19 » Thu, Dec 10, 2020, 7:33 PM UTC

scrollaway wrote: post #136

nojito wrote: post #108

Earlier quoted context omitted.

>AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. This is marketing. It's like saying you want to build a house and the quote you got ends up blowing up 100x overnight. Great example is the 100k credit for startups. You can repeat it's no…

The $100K credit (which I've been granted multiple times) is there because if Amazon can get you to invest serious work into their infra, they'll make up for it in the long run. It's not "lock in", it's sales. The only amazon "lock in" really is their bandwidth-out pricing, which is a sleazy tactic for sure but I'm not hesitant to call it out when it's the case . You can get the $100/$300/$1000 tier if you are in "ju…

> Anyway I don't know what your point is, I'm not even sure if you have one. They're not "marketing" their pricing, nor the fact that you are "forced to design systems that use less resources".

I think they are referring to this statement:

> > AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill.

It is a defense that I've heard in many AWS talks in the past.

Where it turns into a 'marketing' blurb to me is my real world experience in these AWS talks in the places I work. As a real world example, we had a product that required -some- architectural work, but otherwise was solid, and could run on 3 live EC2 instances (2 web LB, 1 live backend) and 1 spare (spare backend)

The Consultant that AWS partnered us with? Suggested a very overdone architectural revamp, moving everything possible into AWS Specific technologies.

It's marketing in that in many of our experiences, we know there is often at least one person on a team who does -not- have the discipline and/or experience to -keep- a system using less resources as the field goes from green to brown.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#139

Post by Retric » Thu, Dec 10, 2020, 7:34 PM UTC

treeman79 wrote: post #119

dvfjsdhgfv wrote: post #96

Earlier quoted context omitted.

> “Sorry, we don’t have the money” is a much better negotiating position than “can we please have our money back?” I agree but why would you like to be in either position anyway? The so-called cloud services are terribly overpriced when compared to traditional servers.

Done correctly they save a lot of IT time. Seem companies hire five 6 figure people to try and cut amazon bill by a couple of grand a month. Never understood spending 50-100k a month to maybe save 5k

It’s often a fixed vs ongoing cost question. Spending 200k to save 5k per month breaks even in 3.4 years.

However, for growing companies that 5k/month AWS premium can hit 200+k/month very quickly

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#140

Post by NegativeLatency » Thu, Dec 10, 2020, 7:35 PM UTC

asciimike wrote: post #133

sandGorgon wrote: post #112

Earlier quoted context omitted.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

Extremely technically, the only GCP product that had this feature was App Engine Standard v1, but looks like it's deprecated as of the end of 2019 ( https://cloud.google.com/appengine/docs/managing-costs#chang... )

Probably hurt revenue ;)