Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

271–280 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#271

Post by benlivengood »

wikibob wrote: post #197
benlivengood wrote: post #155

Earlier quoted context omitted.

It's also a mostly solved problem because advertisers have budgets and it's common to implement globally distributed budget servers to avoid showing more ads than the advertiser paid for, despite tens of thousands of individual web servers needing to know which ads in their inventory have budget left. It's a fun exercise similar to global rate-limiting/load-balancing.

That is fascinating. If you have the time could you (anyone feel free) talk a bit about how you would implement a globally distributed budget? I can imagine a few simple options, but they all seem to have significant shortcomings.

I think the simplest is a tree of servers (which can be sharded by user if necessary for load balancing). The root has the total budget and offers short-term small leases of ad views to child nodes, who may also have child nodes doing the same thing with even smaller leases.

Web servers check with the leaf nodes for every ad they want to show. If that leaf has a budget greater than zero it decrements its own budget and returns success. If the web server gets a success it shows the ad, if not it checks with another budget server or two. Web servers frequently log how many ads were served per client.

Whenever leases are up the intermediate nodes inform the parents of how much was spent and get a new lease. If nodes crash or otherwise don't return their lease then their parents have to assume the whole budget was spent, but leases are kept small to avoid big discrepancies.

If the root crashes then there are problems so the root can be a slow ACID replicated database as long as its immediate children are mostly reliable and take large enough leases to minimize load on the root.

Periodically web server logs are aggregated to adjust the root budgets to account for crashed intermediate nodes and web servers.

The tree approach allows global low latency operation guaranteeing no overspending and minimizing underserving. Nodes are provisioned from the leaves on up to handle the necessary amount of traffic and to ask for leases large enough for 99.X% percent of child requests to succeed.

Any cloud provider could use the same technology on individual hosts to grab leases of CPU, RAM, disk, etc. by the minute per user and terminate services with no budget. Leases could be a lot longer because most budgets are monthly to cover all service needs and not pathological ad campaigns with low budget, high bid, and huge audience.

It's up to cloud (or ad server) providers to decide whether to stop services if the budget system is broken. In most cases it makes sense to fail open and keep serving and eat the loss because shutting everything down will incur even bigger losses.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#272

Post by jyu »

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

There is an easy explanation: It's hard to build this feature, there is no pressing demand from upper management, it's easier to get promoted doing other simpler projects. Think about what a real time snapshot means: you need to know how much of all the services are being used, project that in the future and compute the costs.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#273

Post by dathinab »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

> The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource.

This is simple wrong.

Depending on your use-case disabling active resources is the right reasonable solution with less downsides.

E.g. most (smaller) companies would prefer their miscellaneous (i.e. no core-product) website/app/service to be temporary unavailable then have a massive unexpected cost they might not be able to afort which might literally force them to fire people because they can't pay them....

I mean think about it, what worth is it if my app doesn't go temporary unavailable during it's free trial phase when it means I'm going bankrupt from today to tomorrow and in turn can't benefit from it at all.

Sure hug companies can always throw more money at it and will likely prefer uninterrupted service. But then for every hug company there are hundreds smaller companies which have different priorities.

In the end it should be the users choice, a configuration settings you can set (per preferably per project).

And sure limits should probably be resource limits (like accumulated compute time) and not billing limits as prices might be in flux or dependent on your total resource usage or similar so computing it is non trivial or even impossible.

I often have the feelings that hug companies like Amazone or Google often get so detached from how thinks work for literally every one else (who is not a hug company). That they don't realize that solutions proper for hug companies might not only be sub-optimal but literally crippling bad for medium and small companies.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#275

Post by dathinab »

pstuart wrote: post #259
donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

There are ways it could be done relatively benignly, such as defaulting to paranoid and explicitly opting out. And for those that are heading into that financial barrier it should be a straightforward problem to look at trending to anticipate the shutdown and send out an alert.

Or just ask for a default when opening a new account ;=)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#276

Post by throwclowd »

dathinab wrote: post #263
dijit wrote: post #242

Earlier quoted context omitted.

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

> Note: There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget…

There's something uncanny about understanding the situation enough to turn on the budget alerts, while at the same time not realizing it's not going to help in time if your system runs amok.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#277

Post by seanwilson »

stefan_ wrote: post #251
seanwilson wrote: post #173

https://blog.tomilkieway.com/72k-2/ > To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is…

We've all had a program crash from a stack overflow. The problem seems to be that instead of the "serverless panacea" they were promised, the code they built can now only run on one of many Google servers, none of which are theirs. No way to kick the tires at all. It honestly reminds me of debugging a Jenkins pipeline. Something that was designed to be super generic of a runtime but yet the tooling can inexplicably o…

> We've all had a program crash from a stack overflow.

Launching a cloud function that recursively triggers the same cloud function, that doesn't have a simple safeguard for it looping or blowing up, and where billing scales with the number of cloud functions ticks the "very high risk" and "very high impact" boxes for me. A program running on a single server isn't similar here (you could accidentally create a DoS attack though).

Typical cloud function use is some event gets triggered like a user sign up, the function executes, then it halts. The above isn't a standard use case and is so incredibly risky this approach shouldn't be attempted in my opinion.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#278

Post by dathinab »

brianwawok wrote: post #219
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

There are guard rails in quotas. Like you can only spin up X servers without opening a ticket to ask for more. Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....

They are broken, unreliable, hard to correctly setup guard rails.

I mean like the article mentioned they could have set the instances and concurrency settings to lower values. Which in this case would have worked.

But finding the right settings to balance intentional auto-scaling and limiting auto-scaling to limit of how fast unexpected cost might rise is hard and prone to get wrong.

Let's be honest it's in the end a very flawed workaround which maybe might help (if you know about it, and did it right).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#279

Post by merb »

dijit wrote: post #242
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

thats not the best thing you can do. the best thing you can do is put excessive time into quotas. aws has way better quotas for starters than gcp has, sadly

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#280

Post by vp8989 »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active."

Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period.

The only difference is in that case, it suits AWS to do that whereas in the case of respecting a billing limit, it doesn't.

Post reply on HN