How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#151

Post by minitech » Thu, Aug 02, 2018, 7:40 AM UTC

kompiuter wrote: post #145

ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

Legit question here; what would be a good architecture for this case?

SQL with no ORM. Harder to be unaware of what you’re querying when you have to write the queries.

Not using attempted magic like Firebase would also fix the problem where the home page transfers 9 MB of data from Firebase on top of the 1 MB JavaScript, which appears to be… their entire database or something?? Accessible to the frontend??? Censored excerpt from that response:

  "email":{"stringValue":"soXXXXXXXsu@gmail.com"},"fechaDisponible":{"integerValue":"1527483600000"},"identification":{"nullValue":null},"key":{"stringValue":"1527560309061"},"listaNotificaciones":{"arrayValue":{"values":[{"stringValue":"alXXXXXXXXi@yahoo.com"},{"stringValue":"soXXXXXXXsu@gmail.com"},{"stringValue":"pXXXXX2@hotmail.com"},{"stringValue":"BeXXXXXXXXXXXXXXez@hotmail.com"},{"stringValue":"pXXXXXX0@gmail.com"},{"stringValue":"trXXXXXXXXXXro@gmail.com"},{"stringValue":"joXXXXXXXXXXXXie@hotmail.com"},{"stringValue":"ivXXXXXXXXxd@gmail.

Also appears to expose, for each campaign, the poster’s bank name and date of birth.

And wastes a bunch of various resources making separate requests to a currency conversion service for each amount, as others have noted. And requests /null and /undefined. This might be the most irresponsible development I’ve ever seen.

Re: How we spent $30k in Firebase in less than 72 hours

#152

Post by LolNoGenerics » Thu, Aug 02, 2018, 7:40 AM UTC

thatjsguy wrote: post #63

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

Js is currently more visible than other platforms, also the sheer size of the community takes a role. Idiots are everywhere.

The current trend and curse of DRY and NIH is to solve stuff by adding dependencies and gluing them together. Rookies expect that some software solved the problem at hand without thinking about it. Even worse is that they even apply this to rather simple things. The problem of OP - countig items inefficiently - is absurdly common. IMHO this is the heart of the problem, the new generation is highly uneducated how to handle data.

Re: How we spent $30k in Firebase in less than 72 hours

#153

Post by projektir » Thu, Aug 02, 2018, 7:42 AM UTC

blub wrote: post #129

scrollaway wrote: post #85

Earlier quoted context omitted.

> the most professional cluelessness I’ve ever encountered was in the JS ecosystem. A testament to how low the barrier of entry has gotten. It's both a good and bad thing at the same time. It however leads to having to be ever more so vigilant about at least your first layer of dependencies in that ecosystem, if you do want to be professional. The higher the barrier of entry is to a language, the more likely it is th…

It's been a long time goal of the software dev community to lower barriers to entry through bootcamps and such, in an attempt to "democratize software". And the result is unsurprisingly poorer quality software. So why is it a good thing?

It's a good thing because it avoids the problem of blocking people from entering for largely arbitrary reasons (i.e., country of birth; exposure to computers in childhood).

I don't think the problem of quality should be addressed by arbitrarily axing people from the field. Some sort of a standardization / accreditation seems like a better approach.

Re: How we spent $30k in Firebase in less than 72 hours

#154

Post by TomK32 » Thu, Aug 02, 2018, 7:42 AM UTC

Some have mentioned it here already, but I'd like to emphasis how important application logs are. How much trouble you can prevent by reading and understanding them.

I've seen and fixed such bugs as described in the article, and before you start trying to upgrade anything a look in the log followed by a git bisect session is the first step.

My rails apps have great logs, I get to see what views and partials are rendered, what queries are sent to the database and more important how often all that happens. If the log excerpt for a single request doesn't fit on my screen I know I have to do something.

Re: How we spent $30k in Firebase in less than 72 hours

#155

Post by cpplinuxdude » Thu, Aug 02, 2018, 7:43 AM UTC

Lazare wrote: post #53

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.

Re: How we spent $30k in Firebase in less than 72 hours

#156

Post by TomK32 » Thu, Aug 02, 2018, 7:44 AM UTC

ryandrake wrote: post #56

Definitely looks like several "teach-able moments" here: They learned the hard way about: 1. Developing a fix without understanding root cause (try-something development) 2. Sufficient testing, including load testing, prior to initial deployment 3. Better change control after initial deployment 4. Sufficient testing for changes after initial deployment 5. Rollback ability (Why wasn't that an option?) 6. Crisis manage…

0. shut the app down before it costs even more money.

A money clock on the table isn't fun and if you replace the the app with a landing page and a newsletter form it's completely acceptable for the visitors.

Re: How we spent $30k in Firebase in less than 72 hours

#157

Post by IshKebab » Thu, Aug 02, 2018, 7:46 AM UTC

Lazare wrote: post #53

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happe…

On the other hand I've spent hours debugging an issue in a C++ program I work on only to eventually find that it was a bug in the oldish version of GCC we were using and simply upgrading the compiler would have fixed it.

It's entirely possible that they could have spent hours or days debugging their issue only to find it had already been fixed.

Re: How we spent $30k in Firebase in less than 72 hours

#158

Post by projektir » Thu, Aug 02, 2018, 7:46 AM UTC

blub wrote: post #133

rspeer wrote: post #110

Earlier quoted context omitted.

Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.

Software engineering is very hard and can't be taught in bootcamps, blog posts or on the job. It takes years of concentrated effort and quality learning material. A significant number of graduates of computer science couldn't do software engineering after graduation even before. That barrier to entry is designed to protect society from poor quality software and actual software engineers from having to suffer through…

Where can it be taught, then? There are no resources for this right now, mostly people are left to their own devices.

Most barriers to entry are not designed to protect anyone, they're designed to preserve power. To protect people from bad products, you need regulation, accreditation, etc.

Re: How we spent $30k in Firebase in less than 72 hours

#159

Post by projektir » Thu, Aug 02, 2018, 7:49 AM UTC

onion2k wrote: post #126

r3vrse wrote: post #76

I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to thin…

Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder. This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.

And most companies not knowing how to measure experience, so they pass over experienced devs and can't understand their value...

Re: How we spent $30k in Firebase in less than 72 hours

#160

Post by intothemild » Thu, Aug 02, 2018, 7:49 AM UTC

thatjsguy wrote: post #63

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

Most JS Devs think debugging is just console.log(), that's nice and all, but it's not connecting a debugger, adding breakpoints, and watching the data go around. The amount of times I've caught issues with just a simple game of follow the calls and objects around with the debugger.

As soon as I got to the part where they just upgraded a bunch of libraries.. I rolled my eyes, I was expecting a serious look at something, perhaps even a bug in Firebase or something in-depth. But nope, what we got was "Ooops I didn't think about the number of API/DB calls we were making because we don't think that way, we just assume everything is the fault of the libraries we use."

That kind of attitude is why I cannot wait to abandon JS all together..