Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

81–90 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#81

Post by sushid »

thatjsguy wrote: post #65
whoisjuan wrote: post #59

I find ridiculous that their first solution was to go and upgrade to another Angular version, especially a non-beta version upgrade of a framework that is used in thousands of super high traffic websites with no problems. How clueless can you be?

> How clueless can you be? I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.

Why are you so against junior devs? I've personally never met a junior dev who just starts going ahead and upgrading various dependencies, as you suggest.

If anything, that shows a lack of proper hiring decision on you and your team's part.

I do, however, agree that their practices are horrible (just look at their console, they're console.logging random things, running the dev mode of Firebase, and fetching some USD conversion call 10x on load with no caching) and they're lucky Google bailed them out at the last minute.

Re: How we spent $30k in Firebase in less than 72 hours

#82

Post by nine_k »

Aloha wrote: post #67
ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.

Yes, it's a good investigation tool, but definitely not a fix. That is, if "it started to work now, I don't know why", you're still in trouble, and should continue digging.

Re: How we spent $30k in Firebase in less than 72 hours

#83

Post by thatjsguy »

sushid wrote: post #81
thatjsguy wrote: post #65

Earlier quoted context omitted.

> How clueless can you be? I mean, if you’re junior and you’ve just learned JavaScript, it’s not difficult. I’ve met a lot of monied people who seem to think a junior dev with a few weeks of JavaScript training is equivalent to a senior engineer with a degree. It never works out, at least not for the smart people.

Why are you so against junior devs? I've personally never met a junior dev who just starts going ahead and upgrading various dependencies, as you suggest. If anything, that shows a lack of proper hiring decision on you and your team's part. I do, however, agree that their practices are horrible (just look at their console, they're console.logging random things, running the dev mode of Firebase, and fetching some USD…

> If anything, that shows a lack of proper hiring decision on you and your team's part.

Hey, friend! I had no control over hiring for that gig.

Re: How we spent $30k in Firebase in less than 72 hours

#84

Post by paulddraper »

thatjsguy wrote: post #63
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

It has to do with the barrier to entry.

PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses".

This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.

Re: How we spent $30k in Firebase in less than 72 hours

#85

Post by scrollaway »

thatjsguy wrote: post #63
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

> the most professional cluelessness I’ve ever encountered was in the JS ecosystem.

A testament to how low the barrier of entry has gotten. It's both a good and bad thing at the same time.

It however leads to having to be ever more so vigilant about at least your first layer of dependencies in that ecosystem, if you do want to be professional.

The higher the barrier of entry is to a language, the more likely it is that when you're pulling in dependencies, the code isn't amateurish

Incidentally this is probably why JS juniors think to upgrade dependencies when they encounter unknown situations... a lot of problems in JS do come from your dependencies.

Re: How we spent $30k in Firebase in less than 72 hours

#87

Post by vbezhenar »

everdev wrote: post #4
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.

Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.

Re: How we spent $30k in Firebase in less than 72 hours

#88

Post by newscracker »

It seems like they:

* didn't have good testing

* did no load testing before

* had no code reviews done

* no design reviews done

* had no or very less (useful) application logs

* had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?)

* had no or very little automated tests

* didn't have a detailed post mortem or root cause analysis to see what they could do to prevent it (the ending looked quit amateurish, by pointing to only one thing as a potential lesson)

* wasted a lot of money that could've helped in the future (by instead using that to pay for en error)

If I were in such a team, I would've honestly stated in such an article how deeply ashamed I am that we missed all these things, how "cowboy coding" and heroics must never be glorified, how we got very, very lucky in someone else waiving off charges (this is not a luxury that most startups or one person endeavors would have), and ended with asking for advice on what could be done to improve things (since it's obvious there were many more gaps than just how a few lines of code were written).

To the team that wrote this code and this article — get some software development methodology adopted (any, actually) and some people who can help you follow any of those. Also read the rest of the comments here. You got very, very lucky in this instance. It may not be the same case again, and you may see your "life's work" get killed because you didn't really learn.

Re: How we spent $30k in Firebase in less than 72 hours

#89

Post by thisismydevname »

ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

I don't think they seem aware of the architecture problem, besides they seem proud of having 40 billion request.

Re: How we spent $30k in Firebase in less than 72 hours

#90

Post by yashap »

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

2-7 are sort of understandable for a quick, hacky startup just trying to ship something fast, with minimal experience. But 1 is the really crazy one. Load times spiking to 30 seconds once they start getting significant traffic, and instead of doing a solid investigation, including instrumenting their FE and-or server so they can see where the slowness is (maybe Firebase even includes decent observability by default?), they go all in on upgrading Angular for seemingly no reason? That’s just ... completely illogical.
Post reply on HN