Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

71–80 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#71

Post by boabab »

Aloha wrote: post #67
ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.

updating the framework version is like.. literally the last try-something thing to do though.

Re: How we spent $30k in Firebase in less than 72 hours

#72

Post by megaman22 »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change.

I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, absolutely everything varying.

My head just explodes... I wamt to scream that this isn't how this works, it's not how any of this works.

Re: How we spent $30k in Firebase in less than 72 hours

#73

Post by tgsovlerkhgsel »

heavenlyblue wrote: post #8
everdev wrote: post #4

Earlier quoted context omitted.

Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.

Well, a rogue line could potentially as much shut down your app hosted in DO.

That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill.

When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down.

When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.

Re: How we spent $30k in Firebase in less than 72 hours

#74

Post by whoisjuan »

avargas wrote: post #60
noncoml wrote: post #58

Earlier quoted context omitted.

I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.

A bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.

I agree with you. There are so many ways to make this infinitely more efficient. For starters, why are they re-calculating and re-rendering the value every time they get a new donation? Also, they could store those values in a more cold&cached-storage and just make the reads to update from Firestore. Don't use a freaking database that is charging you for every single read and write, to deal with mundane client-side renderings.

Re: How we spent $30k in Firebase in less than 72 hours

#76

Post by r3vrse »

I spend a fair amount of time on HN.

Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read.

It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window.

Why learn to think critically when you can just 'npm update', fix 37 broken dependencies, and write a blog post about it? Right?

Re: How we spent $30k in Firebase in less than 72 hours

#77

Post by [deleted] »

thatjsguy wrote: post #63
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

[deleted]

Re: How we spent $30k in Firebase in less than 72 hours

#78

Post by hayleox »

This reads less like a "mistake", and more like just not giving any thought to what your code is doing as you write it. A mistake would be if the programmer meant to implement X but instead implemented Y; this sounds more like the programmer just set out to implement Y without even considering other possibilities.

Same for the framework change. An outdated framework might be a second or so slower, but a >30-second load time was never going to be fixed by updating. This is just bad problem-solving skills. When your app is taking 30 seconds to load, you don't just guess at what might make it faster -- you open your JS console and your log files, and you figure out where that time is being spent. Two minutes in the Performance tab of Chrome's developer tools, and you would have figured out the issue was on the back-end rather than the front-end.

Re: How we spent $30k in Firebase in less than 72 hours

#79

Post by diziet »

ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

It's only 460k QPS, with 16k documents everything would be cached really well. Or a single instance of redis can serve that load of reads fairly easily.

Re: How we spent $30k in Firebase in less than 72 hours

#80

Post by tgsovlerkhgsel »

t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

How do pay-as-you-go cloud providers handle DoS attacks (the "smart" ones that simulate a lot of expensive legit-looking traffic, not simple volumetric ones)?

That's the thing that terrifies me. If I'm using S3/Cloud Storage etc., I'm getting charged for each GB of outbound traffic, and I have to assume that the bandwidth available to serve my files is almost infinite.

Post reply on HN