How not to get a $30k bill from Firebase • Hacker News Board

Re: How not to get a $30k bill from Firebase

#91

Post by crankylinuxuser » Mon, May 27, 2019, 2:24 PM UTC

gtsteve wrote: post #73

donmcronald wrote: post #61

Earlier quoted context omitted.

Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.

I guess you could have the SNS topic invoke a lambda that shuts down the specific service if you wanted that but I agree a budgets feature would be nice, especially when provisioning test accounts for developers.

Fine and dandy, but you can't use Lambda in fedramp.

You can't use most aws things in fedramp scope.

Re: How not to get a $30k bill from Firebase

#92

Post by influx » Mon, May 27, 2019, 2:48 PM UTC

donmcronald wrote: post #61

tcgv wrote: post #51

Earlier quoted context omitted.

Amazon provides Billing Alarms for monitoring charges, which is very useful for anticipating unplanned cost surges: - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.

Sounds like the oomkiller. Sounds good in theory until it starts shooting processes in the head that you didn’t intend.

What would you like Amazon to do for stateful services? Should they stop and delete EBS volumes? What about databases? Simply shut them down? What happens when you lose data or it doesn’t come back up?

Re: How not to get a $30k bill from Firebase

#93

Post by bmelton » Mon, May 27, 2019, 2:55 PM UTC

cryptica wrote: post #30

paxys wrote: post #22

Billing problems aside, it's amazing that web development has reached a stage where it's almost too easy to scale a site to 2m active sessions and 20m+ page views in two days with just off-the-shelf tools and barely any specialized skills.

Except for very simple use cases, serverless platforms in general encourage bad software design patterns. For example, in this case, if a regular database was used instead of Firebase, "counting the number of supporters" would have been done inside the database using a query. The worst that the developer could have done is forget to index to the relevant column; performance would have been sub-par but still orders of…

> With Firebase, when doing complex calculations, not only does the data have to leave the database, it often reaches end users

Any time sensitive data reaches untrusted users it is a bad practice, but this needn't be the case on Firebase just because it is serverless.

Firebase cloud functions can be triggered as a response to document saves (or authentications, or a plethora of other things) and run outside userspace, and are available for exactly this sort of work -- keeping the minutiae of upkeep away from the client.

That said, I agree that it's easy to make this kind of mistake with serverless development, but mostly due to lack of existing domain knowledge. It's trivially easy to make similar (or worse) mistakes without a serverless environment by untrained developers, too. It's just a matter of becoming familiar with the toolset, and because serverless tech is newer, fewer are as familiar with them as they are the old things (and there are fewer people to catch those mistakes when they see them.)

Re: How not to get a $30k bill from Firebase

#94

Post by meddlepal » Mon, May 27, 2019, 3:05 PM UTC

influx wrote: post #92

donmcronald wrote: post #61

Earlier quoted context omitted.

Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.

Sounds like the oomkiller. Sounds good in theory until it starts shooting processes in the head that you didn’t intend. What would you like Amazon to do for stateful services? Should they stop and delete EBS volumes? What about databases? Simply shut them down? What happens when you lose data or it doesn’t come back up?

As with all things just having the option would be nice, but it does not mean it is useful or a good idea for everyone to use it.

For personal stuff I would much rather a kill switch than a nasty surprise. For _some_ businesses, that is probably the wrong decision though.

Re: How not to get a $30k bill from Firebase

#95

Post by Gustek » Mon, May 27, 2019, 3:18 PM UTC

@squidrings I believe you are the author?

Very interesting post especially as I am myself currently working on my first firebase based app and making sure I don't get huge bills is one of my worries.

Just little correction needed:

> July last year, a crowd funding campaign went viral in Columbia

It's Colombia not Columbia.

Re: How not to get a $30k bill from Firebase

#96

Post by yjftsjthsd-h » Mon, May 27, 2019, 3:22 PM UTC

kkarakk wrote: post #46

p0llard wrote: post #44

Earlier quoted context omitted.

Why would a junior dev be using _their_ card details? Surely your company has a corporate billing account with Google for this purpose? Perhaps I've misunderstood the situation, but if I were working as a dev at a company and was expected to provide my own card details to pay for APIs, I would walk out the door.

They were just told to make the portal as a learning teambuilding exercise, choice of tools was entirely up to them - i even told the person company would pony up for anything they ended up spending. HR was being a pain about getting card details(it's not easy getting that stuff in india) The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

> The dude bounced off google maps as soon as he saw the form to enter payment info and went straight to open street maps.

Sounds like a happy ending.

Re: How not to get a $30k bill from Firebase

#97

Post by bmelton » Mon, May 27, 2019, 3:30 PM UTC

amelius wrote: post #38

ddffre wrote: post #3

Earlier quoted context omitted.

Glad to hear that Google has done something with it. Also that's one of the reasons I try to use the realtime database, and not firestore. But they still charge for bandwidth there (most if it is consumed by downloading the SSL certificates from the clients).

This must be the first time I hear about Google responding timely and adequately to a specific problem of a customer. Could the publicity have something to do with it?

"If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem."

Re: How not to get a $30k bill from Firebase

#98

Post by EugeneOZ » Mon, May 27, 2019, 3:41 PM UTC

asciimike wrote: post #89

EugeneOZ wrote: post #62

Earlier quoted context omitted.

Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.

App Engine PM: the GAE Daily Spend limits are hard caps that will shut down services when hit. They are different from the broader GCP Billing Alerts which are simply notifications. The issue is that they only apply to certain GAE services (compute, legacy APIs, etc.) and not across the platform.

Thank you for correction!

Re: How not to get a $30k bill from Firebase

#99

Post by faster » Mon, May 27, 2019, 4:04 PM UTC

brianwawok wrote: post #77

donmcronald wrote: post #61

Earlier quoted context omitted.

Alarms are like a garage leaving you a voicemail message before doing $25k of repairs on your car. It’s ridiculous that I can’t set a hard cap on my spending.

On the flip side I would never want my real business to shut off. Hobby project yes! Real business no,

Assuming your revenue and your AWS bill are strongly correlated, I agree. However, when a free service or a bug can increase costs by orders of magnitude without a corresponding increase in revenue your choices are to eat it (not sustainable, for most of us), throttle, or stop. Throttling is hard, so stopping is often the best choice.

Re: How not to get a $30k bill from Firebase

#100

Post by solidasparagus » Mon, May 27, 2019, 6:22 PM UTC

paulgb wrote: post #23

I really wish cloud providers would allow users to set a hard budget that just stops the service if you exceed a threshold. I got a surprise bill from AWS this month (fortunately orders of magnitude less than this, but still ~15x my usual) and am thinking of moving from Lambda to a VPS just so the possibility of this doesn't keep me up at night.

You can set budget alerts, which aren’t as good, but they can let you know if the bill’s going off the tracks. I set up alerts at 50% and 75% of what I’m willing to pay and that’s gotten rid of my ‘unexpected bill’ stress.