Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#381

Post by swyx » Sat, Dec 12, 2020, 8:14 AM UTC

tunesmith wrote: post #303

sudcha wrote: post #267

Earlier quoted context omitted.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

You might want to take another look at your paragraph: > Having been a Googler for ~6.5 years and written dozens of project documents, incident reports, and what not, I knew how to put the case for Google team when they would come back to work in 2 days. That certainly reads as an advantage that most non-Googlers would not have.

it's an advantage, sure, but still he was right in saying he did not use internal connections as alleged. calm down everyone.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#382

Post by 1337shadow » Sat, Dec 12, 2020, 9:05 AM UTC

salmonlogs wrote: post #376

1337shadow wrote: post #285

Earlier quoted context omitted.

Really, "a million other supporting services and all the systems that come with it" ? You get a server with SSH, then you need something expose your container stacks over HTTPS like Traefik (which auto configures), and something for alerting such as Netdata (which auto configures too!), both of which are just a single binary to configure and setup, and probably it won't take long before you have scripts to automate t…

I'm a huge fan of open source and open standards, in fact I always push for and expect portability and avoid proprietary systems where possible. Abstractions like Kubernetes are a fantastic middle ground to provide portability across platforms whilst taking advantage of cloud provider services where they exist. The same for apps and frameworks built on open standards like Kubeflow and Apache Beam. The supporting serv…

I'm not talking about /buying/ a server, but renting one as a service.

99.9% uptime is plenty enough for 99.9% of projects and that's easy to achieve with one server, k8s is not necessary here. You're not concerned with MPLS or whatnot when you rent a server.

I can tell because I'm running governmental websites on this kind of servers actually, with over a thousand admins managing thousands of user requests. I've been deploying my code on servers like that for the last 15 years and it was great really, also got fintech/legaltech project in production and much more.

I guess the project you're describing falls more in the 0.1% of projects than 99.9%.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#383

Post by franciscop » Sat, Dec 12, 2020, 4:15 PM UTC

donavanm wrote: post #239

cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

> "And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource."

Well there's a very easy way, adding a checkbox and an input:

[ ] I am just trying things out, don't charge me more than [ ] USD

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#384

Post by franciscop » Sat, Dec 12, 2020, 4:23 PM UTC

howlgarnish wrote: post #325

vp8989 wrote: post #280

Earlier quoted context omitted.

"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active." Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period. The only difference is in that case, it suits AWS to do that w…

If you hit a rate limit, the marginal portion of requests exceeding that limit is dropped: if you plot the requests, the graph gets clipped. Bad, but not catastrophic. If you hit a billing limit, everything beyond that point is dropped, and the graph of requests plunges to zero . You're effectively hard down in prod.

And for some companies/individuals, if you keep charging then THEY will plunge to a large negative debt. It's not even zero, it's a lot worse than that.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#385

Post by welkie » Sat, Dec 12, 2020, 7:07 PM UTC

asciimike wrote: post #191

NegativeLatency wrote: post #140

Earlier quoted context omitted.

Probably hurt revenue ;)

As a former App Engine PM who spent a lot of time with billing/quotas (though, not the one who deprecated this feature), it's likely due to some combination of: - hard limits caused downtime more often than they prevent these blog posts - hard limits were inconsistently enforced, even within GAE - platform wide quota notifications were implemented (reached "GA"), leaving the question of "how a developer wants to hand…

This is a reasonable way of achieving the balance needed. My company would freak out if we had even a short outage that affected all our customers because we set a billing quota too low. And I'd feel a lot more comfortable experimenting with serverless on my own projects if I knew Google would have my back if I made one of those once-in-a-year mistakes.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#386

Post by user5994461 » Sat, Dec 12, 2020, 7:12 PM UTC

AngusH wrote: post #124

jasonpeacock wrote: post #102

Earlier quoted context omitted.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

But we've had disk quotas before that mostly worked? If anything it seems an easier problem than processor time. I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that. Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails. I don't think they deleted files, it was just you couldn't write more…

>>> But we've had disk quotas before that mostly worked?

AWS has quotas on everything, including quotas on EBS storage per region.

You will realize that after you spin up some instances with disks and it's failing because you've hit 10 TB of EBS storage. Have to raise a ticket to raise the limit.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#387

Post by welkie » Sat, Dec 12, 2020, 7:16 PM UTC

gonzo41 wrote: post #65

onion2k wrote: post #58

Earlier quoted context omitted.

For AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake…

You can trigger events from alarms. And Lambda's only last 15 minutes. So still cheaper than 75K :D.

That's not a bad idea. You could set it up to delete all Lambdas (assuming you've got a CI/CD system capable of redeploying them quickly later) if the billing goes over. Of course, this may hurt you more because of the outage it would cause. Up to you really.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#388

Post by welkie » Sat, Dec 12, 2020, 7:19 PM UTC

donavanm wrote: post #239

cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

> It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource.

This could be ameliorated by using namespacing techniques to separate prod from dev resources. For example, GCP uses projects to namespace your resources. And you can delete everything in a project in one operation that is impossible to fail by just shutting down the project (no "you can't delete x, because y references it" messages).

Aggressive billing alerts and events, that delete services when thresholds are met, could be used only in the development namespace. That way, fun little projects can be shut down and prod traffic can be free to use a bit more billing when it needs to.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#389

Post by welkie » Sat, Dec 12, 2020, 7:27 PM UTC

steren wrote: post #229

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and pr…

> Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and preventing big billing surprises.

This is good to hear. I use Cloud Run a lot for personal projects and I always set concurrency to 80, max instances to 1, memory to 128Mi (unless it's something beefy that needs the memory), and CPU to 1. If I need to scale it up, or I decide to open it up to actual usage, I'll do it when I recognize the need.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#390

Post by welkie » Sat, Dec 12, 2020, 7:29 PM UTC

Retailer wrote: post #340

aidenn0 wrote: post #294

Earlier quoted context omitted.

My guess is that the billing logic is separate from the application logic. There's probably a delay between the two and mostly one-way communication.

1) But they supported this before on GAE. GAE had 'spending limits'. 2) Also if they are able to figure out when you've hit your daily free quota and cut you off almost immediately, how are they not able to figure this out?

If I recall correctly, GAE is an example of something they made specifically to be a cloud product. Products like Compute Engine, GCS, Bigtable, and Pub/Sub are things developed internally and then sold publicly once they realized others might find them useful. Perhaps the products developed first for internal use weren't developed with features like measuring billing usage in real time in mind.