How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#181

Post by blub » Thu, Aug 02, 2018, 10:30 AM UTC

projektir wrote: post #158

blub wrote: post #133

Earlier quoted context omitted.

Software engineering is very hard and can't be taught in bootcamps, blog posts or on the job. It takes years of concentrated effort and quality learning material. A significant number of graduates of computer science couldn't do software engineering after graduation even before. That barrier to entry is designed to protect society from poor quality software and actual software engineers from having to suffer through…

Where can it be taught, then? There are no resources for this right now, mostly people are left to their own devices. Most barriers to entry are not designed to protect anyone, they're designed to preserve power. To protect people from bad products, you need regulation, accreditation, etc.

There are very good universities focusing on computer science and software engineering in most countries.

This is the first step. Then one needs to find a company with a good engineering culture, apply the theory they learned and gather experience. Ideally one should find a qualified engineer as mentor.

Self-study and being aware of developments in the profession are the last piece of the puzzle.

Yes, some people won't be able to do some of these things and as a result they won't be good software engineers. They could still be successful programers, the two aren't necessarily related.

Re: How we spent $30k in Firebase in less than 72 hours

#182

Post by dokalanyi » Thu, Aug 02, 2018, 10:46 AM UTC

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

To be fair I recently upgraded an old PHP install from 5.5 to 7.2 and it sped up the site tremendously. My workflow while troubleshooting is now always going to involve upgrading software.

Re: How we spent $30k in Firebase in less than 72 hours

#183

Post by Oras » Thu, Aug 02, 2018, 11:08 AM UTC

newscracker wrote: post #88

It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed pos…

If they had half of the above they wouldn't use Firebase. Don't get me wrong I think it is a great product and I would use it for MVP but the idea of using Firebase is to get something scalable easily and quickly by focusing on the frontend only. You can tell from their screenshots that they were only focusing on GA metrics and didn't check Firebase console

Re: How we spent $30k in Firebase in less than 72 hours

#184

Post by shady-lady » Thu, Aug 02, 2018, 11:10 AM UTC

Words fail me as to how anybody could put something live without understanding how the application functioned as a whole. Unbelievable.

Re: How we spent $30k in Firebase in less than 72 hours

#185

Post by aidos » Thu, Aug 02, 2018, 11:17 AM UTC

dokalanyi wrote: post #182

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

To be fair I recently upgraded an old PHP install from 5.5 to 7.2 and it sped up the site tremendously. My workflow while troubleshooting is now always going to involve upgrading software.

While upgrading should be an option, just don’t do it as a first step. Changing any dependency comes with risks and moving from version 5.5 to 7.2 of anything screams danger to me.

Sure, there are times when it’s going to work out for you but you should at least have narrowed down your issues before you go down that path.

Re: How we spent $30k in Firebase in less than 72 hours

#186

Post by themarkn » Thu, Aug 02, 2018, 11:19 AM UTC

cloakandswagger wrote: post #49

sitkack wrote: post #40

Earlier quoted context omitted.

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

> My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

They didn’t upgrade packages to solve the mystery billing. They upgraded packages before they checked what was going on with the database. When they saw the high billing that pointed them to the problem and they fixed it.

There was some questionable judgement shown by not checking db requests first, sure, but in no way did someone think “our google billing is high, we better upgrade angular”.

Re: How we spent $30k in Firebase in less than 72 hours

#187

Post by chillacy » Thu, Aug 02, 2018, 11:30 AM UTC

RobertRoberts wrote: post #64

Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt? This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them. This reminds me variable rate mortgages. With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I…

I don’t know what kind of load you expect but web servers don’t just slow down linearly forever: at some point when traffic exceeds capacity, queues fill up, memory maxes out, and all requests grind to a halt with enough load. The way out is scaling horizontally, and you want to do that at some point before you end up sending 503s to the load balancer.

Re: How we spent $30k in Firebase in less than 72 hours

#188

Post by madmulita » Thu, Aug 02, 2018, 11:35 AM UTC

"It is very important that tech teams debug every request to servers before release..." No.

You should know your application's profile, you wrote it.

How many resources does you app need? That's something our developers believe is the "operations team"'s responsability. Well, now that you took the 'devops' role you can no longer keep ignoring this. Your new infrastructure provider will be more than happy to keep adding resources, one can only hope the pockets are deep enough.

With attention to the profile this would have been caught at developing time, maybe testing time.

Re: How we spent $30k in Firebase in less than 72 hours

#189

Post by shusson » Thu, Aug 02, 2018, 11:36 AM UTC

| How we spent $30k in Firebase in less than 72 hours

By not checking Google Billing after you launch your website. At the very least you should have a billing alert.

Re: How we spent $30k in Firebase in less than 72 hours

#190

Post by TeMPOraL » Thu, Aug 02, 2018, 11:37 AM UTC

voltagex_ wrote: post #150

TeMPOraL wrote: post #116

Earlier quoted context omitted.

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is…

>Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything. We've been understaffed for a year and we've been told not to hire any more staff until the new financial year. I've got a "technical debt" item on the backlog but business drives the priorities and…

> Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

Only matters when the fault is located or manifests itself precisely in that layer, and that's always a risk. Consider all the third-party dependencies you use. They usually have only their APIs documented. Fixing a fault usually requires knowledge of the internal implementation.

> We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

> I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

Yeah, I get that. I've seen that. Thing is, you can only play lottery so long with your main product - and also, if your workplace runs an assembly line so tight that you can't spend hours thinking per ticket (excluding the most trivial ones), then something is seriously broken on yet another level.

Ultimately, I guess what I'm saying is that the main problem here is cultural - possibly both on developer and management side. The actual technical tasks aren't usually that challenging.