Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

221–230 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#221

Post by Grazester »

cm2187 wrote: post #146
bartread wrote: post #144

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work togethe…

Except that their core business is to build files on everyone of their users to sell ads... We could have a debate on whether Philip Morris is evil. I am sure most of their employees are pretty decent people.

When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone?

Re: How we spent $30k in Firebase in less than 72 hours

#222

Post by rco8786 »

IshKebab wrote: post #157
Lazare wrote: post #53

Earlier quoted context omitted.

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

On the other hand I've spent hours debugging an issue in a C++ program I work on only to eventually find that it was a bug in the oldish version of GCC we were using and simply upgrading the compiler would have fixed it. It's entirely possible that they could have spent hours or days debugging their issue only to find it had already been fixed.

This isn’t really the other hand. This is saying exactly what OP is. Debug your system and fix the bug when you find it. Don’t randomly change things hoping the problem goes away.

Re: How we spent $30k in Firebase in less than 72 hours

#223

Post by paulddraper »

rspeer wrote: post #110
paulddraper wrote: post #84

Earlier quoted context omitted.

It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.

Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.

> Oh come on

Apologies if I suggested otherwise, but of course programming doesn't need barriers to entry. Just like PCs and the internet don't.

Re: How we spent $30k in Firebase in less than 72 hours

#224

Post by ShaneOG »

magnetic wrote: post #119
ShaneOG wrote: post #23

Earlier quoted context omitted.

A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.

How did you get your Startup credits?

My company was part of a partner acceleration program, however you can apply to be part of the Google Cloud for Startups program here[1], without needing to be in an accelerator.

[1] https://cloud.google.com/developers/startups/

Re: How we spent $30k in Firebase in less than 72 hours

#225

Post by zaarn »

aidos wrote: post #185
dokalanyi wrote: post #182

Earlier quoted context omitted.

To be fair I recently upgraded an old PHP install from 5.5 to 7.2 and it sped up the site tremendously. My workflow while troubleshooting is now always going to involve upgrading software.

While upgrading should be an option, just don’t do it as a first step. Changing any dependency comes with risks and moving from version 5.5 to 7.2 of anything screams danger to me. Sure, there are times when it’s going to work out for you but you should at least have narrowed down your issues before you go down that path.

I mean, it can be fairly safe. When I moved from 5 to 7.0 I simply left the PHP-FPM configuration for 5 around and 5 itself installed. It still is.

I can just spin up PHP5, 7.0, 7.1 in case anything goes wrong without delay.

Re: How we spent $30k in Firebase in less than 72 hours

#226

Post by macspoofing »

overcast wrote: post #215
macspoofing wrote: post #210

Earlier quoted context omitted.

>But acting like that's not a viable solution is silly. Usually you want to understand the problem before solving it. In this case, they wasted a bunch of time doing a bunch of things (upgrading all the dependencies, and refactoring the app) in the hope that something (ANYTHING) they're doing hopefully fixes a problem they don't understand. Smart move?

Everyone can sit back in hindsight and act like armchair developers. When you're in the shit, sometimes you make not so smart moves, now they've learned from their mistakes. Which is why they are talking about it.

>Everyone can sit back in hindsight and act like armchair developers.

That's one perspective. But come on! I really don't understand that attitude that when presented with a problem, the first approach is to spend a few days blindingly refactoring code and upgrading all the underlying frameworks. Seriously?

The problem is also obvious if you just stop and think about it for a second:

- They are using Firebase. For the purposes of diagnosing our issue we can assume the backend will scale well (for trivial queries) and the pipe between server and client should be wide. Firebase could be the problem, but odds are Firebase didn't go down on you just as your go-live went ahead.

- Because they are using Firebase, their app is completely client-side.

You can go through the potential areas of concern:

1) UI has trouble rendering. That should largely be independent of the number of users. If this was only a UI issue you'd expect some users to have problems (maybe ones that created a large amount of artifacts) but not all users. Presumably before going live, the app worked well with their test datasets.

2) Some combination of UI or Network or Data model. They noticed their web-app got slower as the number of users grew. So question is why would individual user session slow down as the total number of users grow? It must be that a single-user view is somehow dependent on the total number of users in the system. WHY?!? We know Firebase is fast, but any fast system can choke if you have a bad data model. So it could be a slow query. Or it could be a too large of a response being sent down (again, why would a large response be sent down). Maybe it was a huge json object and the UI locked up. Or something like this.

It really shouldn't have taken long to at least target potential areas to explore. HELL, you should be able to see the issue immediately if you open up the network tab. You'll see which requests are either taking forever, or lead to large amount of data being transferred or both.

It really isn't about 'armchair developers'. I've been in situations where things are falling apart and you need to figure shit out. Our product is on-prem and used in hospitals and is connected to multitudes of other systems controlled by other vendors. When you're trying to diagnose issues, you have to have a rational approach based on some reasonable hypothesis.

Re: How we spent $30k in Firebase in less than 72 hours

#227

Post by mirimir »

che_shirecat wrote: post #55
mirimir wrote: post #38

Earlier quoted context omitted.

Ah, I didn't look closely at the charges graph. So yes, it maxed at ~5000 USD per day. I gotta say that using "$" for both USD and COP is confusing. So you must say "USD $x" and "COP $x". Then why bother with the "$"?

there's over 20 different types of "dollars"

That's really my point. Not Americanism or whatever.

I mean we have kilograms, meters and seconds. And they're the same for every country.

But "$" (dollars and other currency units) means different things, depending on the context. Similarly for ounces, pounds, feet, gallons, etc. So you're left with constructions like "US $" or "USD" or "USD $" vs "Can $" or "CAD" or "CAD $". Just as with "avoirdupois ounce" vs "troy ounce", "US gallon" vs "imperial gallon", and so on.

So anyway, I always write "foo USD", "foo EUR", "foo mBTC" and so on. To avoid ambiguity.

Re: How we spent $30k in Firebase in less than 72 hours

#229

Post by cm2187 »

Grazester wrote: post #221
cm2187 wrote: post #146

Earlier quoted context omitted.

Except that their core business is to build files on everyone of their users to sell ads... We could have a debate on whether Philip Morris is evil. I am sure most of their employees are pretty decent people.

When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone?

> When was the last time the consumption of a Google ad lead to cancer and the eventual death of anyone?

Great logic. Philip Morris is evil because they sell products that cause cancer and death. Therefore Equifax, who extorts money from people to protect them against Equifax polluting their credit rating, and who leaks their data into the wild is not evil, because Equifax doesn't cause cancer nor death!

Re: How we spent $30k in Firebase in less than 72 hours

#230

Post by ocdtrekkie »

gcbirzan wrote: post #198
boulos wrote: post #137

Earlier quoted context omitted.

Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :). Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”. In this case, they even…

That, or, as explained in the SRE book, the extra load should've been shed. The binary choice is not always the right one. Though, granted, in this case, it wouldn't have mattered...

Yeah, I think "the web server ran out of money at 3 PM today" isn't the right choice, whereas "a lot of people had a hard time getting to the site when it got HN'd around 3 PM, but it was back to normal by 5 PM" is a better result.

For smaller organizations, something being down during extreme load is a recoverable problem, but owing the cloud provider all of their money may not be. (Note that even in the case here where Google got them the grant to cover this bill, this is still probably 35K in grant money that could've gotten them further or been used better elsewhere.)

Post reply on HN