Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

61–70 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#61

Post by mirimir »

ithkuil wrote: post #48
mirimir wrote: post #38

Earlier quoted context omitted.

Ah, I didn't look closely at the charges graph. So yes, it maxed at ~5000 USD per day. I gotta say that using "$" for both USD and COP is confusing. So you must say "USD $x" and "COP $x". Then why bother with the "$"?

apparently the $ sign has its origins in the spanish Peso, where the p and s were gradually being merged together in abbreviations. futhermore the US Dollar itself stems from the Spanish Dollar: "The U.S. dollar was directly based on the Spanish Milled Dollar when, in the Coinage Act of 1792, the first Mint Act, its value was fixed [..] as being "of the value of a Spanish milled dollar as the same is now current

True enough.

But it's still potentially confusing, when stuff gets translated with no context for currency values. Especially, I imagine, if you don't know either Spanish or English.

Re: How we spent $30k in Firebase in less than 72 hours

#62

Post by pishpash »

"Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies."

Doesn't sound like a future great company to me, especially when their lesson from this was Google will bail them out and "It is very important that tech teams debug every request to servers before release." rather than hiring less cavalier employees and putting in better process.

Re: How we spent $30k in Firebase in less than 72 hours

#63

Post by thatjsguy »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refactoring was also pretty popular, although it usually just ended up making things more complicated.

It’s silly to suggest that JavaScript itself is somehow responsible for this. It’s obvioisly just a tool. But I have to say, the most professional cluelessness I’ve ever encountered was in the JS ecosystem.

Re: How we spent $30k in Firebase in less than 72 hours

#64

Post by RobertRoberts »

Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt?

This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them.

This reminds me variable rate mortgages.

With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I really don't understand the cloud at all for such high cost resources with literally unlimited/unpredictable pricing.

Can someone explain this risk/reward scenario here?

Re: How we spent $30k in Firebase in less than 72 hours

#65

Post by thatjsguy »

whoisjuan wrote: post #59

I find ridiculous that their first solution was to go and upgrade to another Angular version, especially a non-beta version upgrade of a framework that is used in thousands of super high traffic websites with no problems. How clueless can you be?

> How clueless can you be?

I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.

Re: How we spent $30k in Firebase in less than 72 hours

#66

Post by ct520 »

thatjsguy wrote: post #63
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

I concur. I’m having a hard time not coming off as a dick while trying to opine on how amateurish I think this is. I like JS but the devs and ecosystem leaves me wanting.

Re: How we spent $30k in Firebase in less than 72 hours

#67

Post by Aloha »

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.

Re: How we spent $30k in Firebase in less than 72 hours

#68

Post by badlucklottery »

paranoidrobot wrote: post #21
iamgopal wrote: post #16

Earlier quoted context omitted.

So basically we want to have cake and eat it too.

This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.

>that the billing and billing limits should also be able to respond similarly

How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do.

Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app ("Sorry, we're not taking new users at the moment" or the like).

Re: How we spent $30k in Firebase in less than 72 hours

#69

Post by wmij »

thatjsguy wrote: post #65
whoisjuan wrote: post #59

I find ridiculous that their first solution was to go and upgrade to another Angular version, especially a non-beta version upgrade of a framework that is used in thousands of super high traffic websites with no problems. How clueless can you be?

> How clueless can you be? I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.

Isn't this why you do pull requests, code reviews and QA before deploying? You need to make sure that the more senior(s) on the team and proper testing catch these things before deploying.

Re: How we spent $30k in Firebase in less than 72 hours

#70

Post by [deleted] »

ithkuil wrote: post #48
mirimir wrote: post #38

Earlier quoted context omitted.

Ah, I didn't look closely at the charges graph. So yes, it maxed at ~5000 USD per day. I gotta say that using "$" for both USD and COP is confusing. So you must say "USD $x" and "COP $x". Then why bother with the "$"?

apparently the $ sign has its origins in the spanish Peso, where the p and s were gradually being merged together in abbreviations. futhermore the US Dollar itself stems from the Spanish Dollar: "The U.S. dollar was directly based on the Spanish Milled Dollar when, in the Coinage Act of 1792, the first Mint Act, its value was fixed [..] as being "of the value of a Spanish milled dollar as the same is now current

[deleted]
Post reply on HN