So when in IT industry there will be responsibility for what we engineer?
Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
201–210 of 397 posts
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#202Earlier quoted context omitted.
It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with this one weird trick").
It's really not a simple problem because the next action depends on the choice the developer wants to make: do they increase the budget or decrease usage, and no cloud provider wants to make this choice because no matter what the choice is it will be viewed as wrong. The best they can do is provide developers the best insight and tooling to make this choice themselves.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#203Earlier quoted context omitted.
It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
If the amount of storage that you can use is limited by quota (say 50GB) the problem becomes relatively easier.
You set a quota for 50GB of storage and no more. The server then restricts you by disk quota to that amount of storage.
The cost is then calculated as 1.15USD per month.
So you don't pay more than 1.15 per month.
Compute and transfer (and other things) could be covered by separate similar quotas with a single maximum spend figure at the bottom of the table.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#204happened to me something very similar, I was using cloud run to fetch some subreddit posts and ended with a "recursive" way, because of that, billions of invocations was made... luckly, I was at the front of the computer and stopped before, but the bill was around $4000! I contacted the google support and explained all, they "forgot" my debit because of the bug
So basically no one's testing their code anymore and just throws it into a paid service?
Great that Google was so lenient and all, but I really don't get the appeal of using a hyper-scaler when a VM with docker support can be setup in literally seconds and on-demand pricing of less than 10 cents/hour for most quick-and-dirty tasks.
Am I missing something here?
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#205Earlier quoted context omitted.
So you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?
If I'm the only developer on a project and I really need to get to market I might do just that. I sometimes do day hikes on weeknights so this is actually a likely scenario for me.
Do you go hiking alone without your phone? That seems dangerous.
And why would you start a test if you won't be there to see the results of the test? Seems more sensible to either leave after you've run the test or wait to do so until you get back.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#206As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…
I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit.
It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#207writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…
And this comment makes you look incredibly naive and narrow minded.
Running some code on a CPU != running a startup. Great you can buy a Dell server on eBay, or you can build a powerful desktop, or rent a VM or get a droplet or scrape on lowendbox. These are not a secret and there is a great reason no one does this other than hobbyists and neckbeards.
You do your testing and it works, then what? You have to deliver scalable reliable systems in production that require identity management, security, backups, resiliency, reliability, various networking services and a million other supporting services and all the systems that come with it. Never-mind actually scaling the application, monitoring it and all the tools, systems and processes needed to run reliable systems in production.
The eBay servers provide you exactly zero of that and you've just wasted time setting up an environment that is a snowflake and doesn't represent reality. Testing on the cloud on exactly the same platform you would use for production has a lot of benefits when you look at value as limited developer time delivering value to customers and the business.
Whilst the $1200 server on eBay might be cheap today, you are entirely missing the hidden cost of lost time when your team of developers costing $M/year are wasting on testing in an environment that doesn't help them find and solve production issues. You don't need many hours of wasted time or downtime to lose all of your so called cost gains.
Optimising for absolute minimum cost is a fools errand that only slows down actually delivering production systems that deliver value to your customers.
Please spend some time thinking bigger about the opportunity cost and value delivery of technology beyond the immediate dollars and cents - it might surprise you.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#208Earlier quoted context omitted.
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with…
You need separate costed quotas for each type of activity
with a combined total at the bottom.
You could also have a setting in the admin panel as to what the system should do:
[ ] I want to keep going beyond my quotas (but email me)
[ ] Please shutdown my site
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#209Earlier quoted context omitted.
One of my favorite quotes of all time. J. Paul Getty was quite the weirdo. His Wikipedia article is worth a look, especially the section on his frugality.
Lol. I love it. I moved to a state I'd never considered because it had the largest, cheapest building in the US. It's 220,000 square feet, but I've lived in a tent out back for the last 6 months because I can't get an occupancy permit, it's not zoned residential, and I refuse to pay rent on an apartment.
You live in a 220,000 square foot building?! Is this an abandoned missile silo or something? I want to know more.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#210Thank you for sharing. I was actually thinking of using fire base for my project. They make us so easy to sign up for free tier. Awaiting to see what happens in part 2
I recently put the (soft) kibosh on a project in my stable trying to switch to FireBase at the last minute. It looks attractive but the business aspects are frankly frightening, and I'm not even talking about the risk of a large bill. Getting your metrics 24h late sounds like a deal killer for me. So much for observability!
Minor nit: many non-billing metrics are near-real time, e.g. DB concurrents, cloud functions CPU/RAM usage; any metrics that require aggregation (storage, billing) are going to be batched less frequently. This is going to be true across all platforms of non-trivial scale (eventual consistency + batch jobs).
Second note: the number of people who actually do this is very low (a few a year, of hundreds of thousands of developers). The blog posts are scary, but in my ~five years at Firebase, I'm pretty sure we refunded every one. As my boss (James Tamplin, CEO of Firebase) used to say, "There are lots of bad systems, but rarely are there bad people."