Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

191–200 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#191

Post by purerandomness »

magnetic wrote: post #119
ShaneOG wrote: post #23

Earlier quoted context omitted.

A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.

How did you get your Startup credits?

Not OP, but when our startup was accepted into https://www.thefamily.co/ (which is basically a European YC), amongst other perks, we got a huge amount of AWS credits. I'm sure other incubators/accelerators do similar offers for GCE/AWS.

Re: How we spent $30k in Firebase in less than 72 hours

#192

Post by deadbunny »

cheeze wrote: post #91
itcmcgrath wrote: post #20

Earlier quoted context omitted.

Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.

This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly. Its a problem with most cloud providers, but Google seems to be notorious for it.

In my experience none of those folks are paying for support.

Re: How we spent $30k in Firebase in less than 72 hours

#193

Post by bananaboy »

IshKebab wrote: post #157
Lazare wrote: post #53

Earlier quoted context omitted.

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

On the other hand I've spent hours debugging an issue in a C++ program I work on only to eventually find that it was a bug in the oldish version of GCC we were using and simply upgrading the compiler would have fixed it. It's entirely possible that they could have spent hours or days debugging their issue only to find it had already been fixed.

But after those hours you knew exactly what the problem was and exactly what the solution was. You were in a much better position and could make better judgement calls at that point. It's better to dive in and figure out a problem than to just randomly upgrade things.

Re: How we spent $30k in Firebase in less than 72 hours

#194

Post by Cthulhu_ »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Yeah I don't get it either, if you have performance problems you look at the network tab and see what's up, instead of yolo-upgrading your framework with fingers crossed and hope it works. Measure before fix.

Re: How we spent $30k in Firebase in less than 72 hours

#195

Post by megous »

newscracker wrote: post #88

It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed pos…

They just didn't profile before optimizing. That's all.

Re: How we spent $30k in Firebase in less than 72 hours

#196

Post by weavie »

thatjsguy wrote: post #63
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

> Refactoring was also pretty popular, although it usually just ended up making things more complicated.

Yeah I'm guilty of this one. Sometimes you know the problem is somewhere in a particular area of code, but that code is all over the place. Pulling it apart and refactoring it can be a good way of understanding all its dependencies. If the refactoring doesn't help, just don't check it in..

Re: How we spent $30k in Firebase in less than 72 hours

#197

Post by zwily »

bufferoverflow wrote: post #130
tgsovlerkhgsel wrote: post #80

Earlier quoted context omitted.

How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)? That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.

You should rate-limit and cache the potentially expensive paths of your app. There's no real way to protect against a clever DDoS, someone will have to eat the costs.

You can’t really rate limit server-side with Firebase. If a client has read-permission on a data node, an attacker can just read from there as many times as they want.

There are some ways to limit writing using the rules engine but you’ll still get charged for failed writes. :)

Only real way to rate limit firebase that I know of is to put some sort of proxy service in between, but then you lose a lot of the advantage of using FB in the first place.

Re: How we spent $30k in Firebase in less than 72 hours

#198

Post by gcbirzan »

boulos wrote: post #137
ocdtrekkie wrote: post #132

Earlier quoted context omitted.

Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next w…

Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :). Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”. In this case, they even…

That, or, as explained in the SRE book, the extra load should've been shed. The binary choice is not always the right one. Though, granted, in this case, it wouldn't have mattered...

Re: How we spent $30k in Firebase in less than 72 hours

#199

Post by Humdeee »

cheeze wrote: post #91
itcmcgrath wrote: post #20

Earlier quoted context omitted.

Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.

This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly. Its a problem with most cloud providers, but Google seems to be notorious for it.

It's quite the phenomenon. Reaching out via social media to garner a company's attention when all other avenues have been exhausted... what a world we live in.

"Company ignoring you? Send out a tweet, that'll work!"

And it blows my mind that it actually does. It's very sad.

Re: How we spent $30k in Firebase in less than 72 hours

#200

Post by overcast »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

There has been MANY times, upgrading outdated libraries or software versions resolved similar performance issues for me. Should you just immediately run update all? No. But acting like that's not a viable solution is silly.
Post reply on HN