Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

51–60 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#51

Post by donmcronald »

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

This is my worst nightmare. Lol. I guess now is a great time to give Azure a shoutout for sitting on their hands for 8 years without so much as a response to the community for half a decade [1].

At least AWS allows using a prepaid credit card so they’ll need to call me if things go haywire. I bet if that $72k charge went through it would have been much harder to get out of. “Sorry, we don’t have the money” is a much better negotiating position than “can we please have our money back?”

1. https://feedback.azure.com/forums/170030-signup-and-billing/...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#52

Post by jedimastert »

walrus01 wrote: post #30
JimDabell wrote: post #24

Earlier quoted context omitted.

> this post quite honestly just makes me smirk. It seems pretty callous to laugh at somebody else’s $72K misfortune, especially as they took reasonable steps to set a budget on the platform.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again. It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. It's not callous - it's the p…

> It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring.

You're not really helping your argument here. Particularly if people have been attacked for over a decade and no one has put up a "aggressive geese" sign

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#53

Post by jcelerier »

LeonM wrote: post #39
jcelerier wrote: post #29

Earlier quoted context omitted.

Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year

You confuse cost with salary. Someone who makes 35k a year costs much more. Think about office space, training, insurance, payroll costs etc.

35k is the gross which encompasses a big part of that except office space.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#54

Post by bjarneh »

spacemanmatt wrote: post #26
daneel_w wrote: post #14

It's "fantastic" how Google by the end of the article still come out as a good and friendly bunch...

Author is an ex-Googler so he knew exactly how to speak to the system

I'm sure that was an important factor, but wouldn't this in any case just be a bill to a startup with no money on hand? It's hard to make companies (without money) economically responsible for anything I guess, it even seems hard to make companies with money responsible sometimes.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#55

Post by that_guy_iain »

daneel_w wrote: post #14

It's "fantastic" how Google by the end of the article still come out as a good and friendly bunch...

Well, when someone lets you off a 72k bill you generally think nicely of them. But considering Google had no way of collecting that and asking for it would have resulted in loss of other business (they'll keep hosting there and keep paying them) and this isn't like Google lost 72k or like 72k even matters to them so it's just good PR, good business to get money on the backend and faster.

I have to wonder if even if they tried to get the money would they legally been able to fight it. From my experience with judges in Europe, they would have most likely looked at the budget being ignore and then upgrading someone from a free to a paid plan without consent and told Google it was their own fault and the services weren't ordered or authorised.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#56

Post by teekert »

MaxBarraclough wrote: post #10
bharatsb wrote: post #4

Part 2: https://blog.tomilkieway.com/72k-2/

> Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.

Yeah a friend of mine wanted a real cert, not letsencrypt (I don't understand how that is more real but ok), as a bit of a noob he clicked around on the AWS website and some days later had a bill op 1500 eur. They also nulled it. Still, this scares the hell out of me.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#57

Post by walrus01 »

jedimastert wrote: post #52
walrus01 wrote: post #30

Earlier quoted context omitted.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again. It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. It's not callous - it's the p…

> It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. You're not really helping your argument here. Particularly if people have been attacked for over a decade and no one has put up a "aggressive geese" sign

Quite literally in my specific area, the former is true, and also it's a fact the city government has put up a number of signs in the nesting area. Still happens.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#58

Post by onion2k »

gonzo41 wrote: post #47
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars. If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a r…

For AWS you get about $0.20 per 1 million requests on Lambda.

If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article.

But really the first lesson you should learn in any cloud setup is Billing Alarms

Alarms only tell you that something is going wrong. They don't stop it. If your mistake is costing $1000/minute and you're an hour away from a computer you have a very expensive problem.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#59

Post by donmcronald »

ZephyrBlu wrote: post #49
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Probably because it's not so simple on the backend. I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update. Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time. It would be a great feature, but I can imagine it being very complex. It's als…

Azure has it for some plans [1], but not others like pay-as-you-go. It seems arbitrary.

1. https://azure.microsoft.com/en-us/support/legal/offer-detail...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#60

Post by scrollaway »

LordHeini wrote: post #42

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone.

And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and flatter prices.

When you design your application with AWS, pricing has to enter your design considerations. For example if you are designing something that will interact a lot with S3 you want to minimize PUTs. You want to minimize ram usage on lambda by streaming rather than buffering. Etc.

AWS is not a suitable product for playground stuff. The only reason it gets used as such is because it's easier if you're already using AWS for other things (or it's you're already very familiar with it).

Post reply on HN