The blog authors should state up front that Google covered the costs with an additional startup grant, and that this was entirely due to a quadratically expensive query.
How we spent $30k in Firebase in less than 72 hours
161–170 of 249 posts
Re: How we spent $30k in Firebase in less than 72 hours
#162A lot of people here talks about lack of load testing and other "do it the right way" type of advice, but remember this is a startup. In my opinion, solid testing foundation will be such an overkill and the time is better spent on implementing more features. Also I bet they did some manual testing. They didn't catch it because this latency can only be seen by the account with a lot of followers. I agree that their fi…
Lack of testing and poor code quality are completely different things though.
Re: How we spent $30k in Firebase in less than 72 hours
#163Earlier quoted context omitted.
I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…
So I completely agree, but let me play devil's advocate. If you do go and track down the problem in your depedency and file a bug, one of two things is likely to happen: they close it and say it's fixed in the latest version or they refuse to accept your bug because it's filed against an old version. Skipping the track it down part and just jumping into upgrading can be a time saver. It works fairly well if you fit i…
There is a possibility you found a bug and a workaround is known/suggested
There is the possibility you are told you made a mistake in thinking it’s a bug with the library
Re: How we spent $30k in Firebase in less than 72 hours
#164Earlier quoted context omitted.
> the most professional cluelessness I’ve ever encountered was in the JS ecosystem. A testament to how low the barrier of entry has gotten. It's both a good and bad thing at the same time. It however leads to having to be ever more so vigilant about at least your first layer of dependencies in that ecosystem, if you do want to be professional. The higher the barrier of entry is to a language, the more likely it is th…
It's been a long time goal of the software dev community to lower barriers to entry through bootcamps and such, in an attempt to "democratize software". And the result is unsurprisingly poorer quality software. So why is it a good thing?
Because people think bad software is better than no software
Re: How we spent $30k in Firebase in less than 72 hours
#165Earlier quoted context omitted.
I wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.
A bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.
Changing the currency will re-trigger all those conversion calls ...
Re: How we spent $30k in Firebase in less than 72 hours
#166Earlier quoted context omitted.
Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.
This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly. Its a problem with most cloud providers, but Google seems to be notorious for it.
From my experience Firebase are different. They're now a part of Google but support has been excellent anytime I needed it - both before and after they were acquired.
Re: How we spent $30k in Firebase in less than 72 hours
#167Earlier quoted context omitted.
Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, a…
Unfortunately I often see developers asking for complex diagnose in urgent emergencies. When it's literally like a 30 minutes reaction time frame. So from business pov there should be first quick and dirty unperfect analysis which is necessary for emergency decisions. Sometimes you can just prolong caching time as quick fix, but sometimes there is no such option. Programmers like nice and calm working environment but…
The first thing I'm likely to say to my team when an outage occurs is something along the lines of "stop the bleeding". That might mean bypassing the affected service if possible, rolling back a recent release, or reducing the amount of traffic (we're fortunate enough to have most of our traffic coming from sources we can throw a kill switch on).
However we go about it, the first priority is to give ourselves some space to properly analyse the issue and find the real solution without the rest of the business worrying loudly about things being broken.
Re: How we spent $30k in Firebase in less than 72 hours
#168I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to thin…
Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder. This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.
It comes with experience, but also with more senior devs explaining it to you.
Seing someone use firebase to save paiements, then recompute a total from a collection, and as a consequence having its system explodes with less than one session per second, means everybody in the team drank the « let’s use this nosql google shiny tech, it’s so cool » cool aid.
Even one conversation with any senior dev having some kind of experience with backend development would have asked about expected load, types of queries, data model, etc. And concluded that storing paiements was probably the least interesting scenario for using a tech like firebase.
Re: How we spent $30k in Firebase in less than 72 hours
#169So it's a classic N+1 query?
Not really. N+1 is about « to many » relationships. You would need at least 2 collections to run into that problem.
In sql equivalent, what happened would be more like doing a full table scan on each query instead of using an index ( and not even that, because pre-computing a total isn’t really like an index)
This kind of « account balance » problems are typical of the problems where transactions are really useful. But also historically the kind of problem where nosql techs do a poor job ( they’re more built with « eventual consistency » in mind than atomic or transactional behaviors)
Re: How we spent $30k in Firebase in less than 72 hours
#170Earlier quoted context omitted.
> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is…
There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.
There's a good chance this is true, but this can happen even with competent developers. One person writes the unoptimized function figuring it can be improved later, someone else assumes this must be cheap and drops it on the front page. Or there was meant to be a caching layer, that didn't work because of some bug.