Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

171–180 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#171

Post by tdumitrescu »

cpplinuxdude wrote: post #155
Lazare wrote: post #53

Earlier quoted context omitted.

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.

It's only about finding needles in haystacks if you don't have sufficient instrumentation and monitoring in place. For most production issues your tools should be able to guide you at least the first 75% of the way to your issue, even if they can't usually offer a good fix (though sometimes they can, such as when new relic points out missing db indices causing unnecessary full table scans).

Re: How we spent $30k in Firebase in less than 72 hours

#172

Post by bsaul »

wiradikusuma wrote: post #99

If there's one thing Google App Engine has taught me, over the course of 5+ years using it: Denormalize your data (or, optimize for read).

Just a precision in case a junior dev read this comment : denormalizing has nothing to do with the problem experienced by the OP. They just queried the whole collection of paiements instead of precomputing a total everytime a new paiement arrived.

Denormalizing would have required having a relation between collections. Here there was just one (so there was nothing to denormalize).

Optimizing for reads ( or simply think about read performance) doesn’t require denormalizing. It can also be a matter of creating an index, or precomputing values in a cache, like in OP case.

Re: How we spent $30k in Firebase in less than 72 hours

#175

Post by bshimmin »

curun1r wrote: post #121
coding123 wrote: post #109

Earlier quoted context omitted.

Are you... in the JS ecosystem yourself? I used to be in the Java ecosystem, the C# ecosystem, the PHP ecosystem... and I could have made the statement "the most professional cluelessness I’ve ever encountered was in the X ecosystem." I think it's just an industry thing.

Having spent more than a decade working and hiring in the Java world, I'm inclined to agree. Having spent much of the past year writing Rust and interacting with that community, I'm inclined to disagree. My overall feeling is that it isn't only JavaScript, but it is JavaScript and languages that attract a higher caliber of developer don't suffer from this problem the way that languages that appeal to a wider demograp…

I feel like JavaScript is a bit of a special case, because there are people out there who, 5-10 years ago, did HTML, CSS, and jQuery, and called themselves "front-end developers", and then along came Node, so they read a few tutorials because it was just JavaScript, and they got some stuff vaguely working and then rebranded themselves as "full-stack developers" - throw in Firebase and Heroku and you have people who can store data and deploy applications to the web with only the most superficial understanding of what they're doing (and yes, that's how you end up spending $30K accidentally on Firebase and thinking that upgrading Angular might fix the problem).

Re: How we spent $30k in Firebase in less than 72 hours

#176

Post by iamforreal »

rafael859 wrote: post #148
bartread wrote: post #144

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work togethe…

I agree with your argument, but I don't think that people are calling Google evil because they retired some products. See [0], which was resolved (I think, though I can't find a source for that) due to the fact that Google is comprised of a lot of different people, and [1], which is a fairly recent announcement that has made some people uncomfortable. [0] https://news.ycombinator.com/item?id=17202179 [1] https://news…

For me, I find what they're doing with android pretty evil (disallowing manufacturers from also offering phones with alternate distros/os')

Re: How we spent $30k in Firebase in less than 72 hours

#177

Post by fauigerzigerk »

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

0. Choose a platform that doesn't bankrupt you if you're less than perfect at 1 to 7.

Re: How we spent $30k in Firebase in less than 72 hours

#178

Post by nevi-me »

cpplinuxdude wrote: post #155
Lazare wrote: post #53

Earlier quoted context omitted.

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.

I agree. It's also about trusting developers' intuition when debugging problems. We recently (past few working days) went through something similar; where we had a problem blocking us from a release, and people scrambling to figure out the problem.

We have some software that was returning different results from different environments, and we couldn't figure out the problem. There was a lot of panic in the room, from upgrading and downgrading Maven dependencies, building things inside and outside of Jenkins, and all sorts of random things.

We kept telling the project leadership that we're poking at the wrong part (intuitively), but they kept pushing. I've had to explain how Maven works, how building on Jenkins doesn't differ to building from our IDE's, etc.

It's only when we asked for isolation from the (human) elements, that we had the freedom to properly debug.

In the end, an unstable sort was the cause of the issue. We were taking the last element from an array, but not sorting the array first.

All of the stuff we did since last Thursday to Tuesday evening didn't help us.

So, I agree, you need good humans who are good at responding well when things break.

Re: How we spent $30k in Firebase in less than 72 hours

#179

Post by radicalbyte »

ct520 wrote: post #66
thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

I concur. I’m having a hard time not coming off as a dick while trying to opine on how amateurish I think this is. I like JS but the devs and ecosystem leaves me wanting.

The sad thing is that this is in my experience the norm. Not just in frontend/js either. It's everywhere.

Re: How we spent $30k in Firebase in less than 72 hours

#180

Post by blub »

projektir wrote: post #153
blub wrote: post #129

Earlier quoted context omitted.

It's been a long time goal of the software dev community to lower barriers to entry through bootcamps and such, in an attempt to "democratize software". And the result is unsurprisingly poorer quality software. So why is it a good thing?

It's a good thing because it avoids the problem of blocking people from entering for largely arbitrary reasons (i.e., country of birth; exposure to computers in childhood). I don't think the problem of quality should be addressed by arbitrarily axing people from the field. Some sort of a standardization / accreditation seems like a better approach.

The country of birth and exposure to computers in childhood are not preventing people from becoming software engineers. I'm living proof of that, as are many people all around the world.

And axing is not arbitrary, it's generally done based on experience and know-how. Not everyone can or should become a software engineer.

Post reply on HN