How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#241

Post by foxylad » Thu, Aug 02, 2018, 9:46 PM UTC

thatjsguy wrote: post #63

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

r/Javascript/PHP/g # 7 years ago

r/Javascript/Visual Basic/g # 14 years ago

r/Javascript/Basic/g # 21 years ago

Us old/wise/thoughtful folk have denigrated the tools that young/foolish/impetuous kids use since we were they.

We need both: yes, these young people made some mistakes, but I'm in awe at what they achieved. They built, triaged and fixed a massively successful campaign in the time I would have taken scoping out the requirements. Oh, and gladhandled Google into paying the tab... impressive!

[Update: formatting]

Re: How we spent $30k in Firebase in less than 72 hours

#242

Post by fouc » Thu, Aug 02, 2018, 10:45 PM UTC

nevi-me wrote: post #178

cpplinuxdude wrote: post #155

Earlier quoted context omitted.

That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.

I agree. It's also about trusting developers' intuition when debugging problems. We recently (past few working days) went through something similar; where we had a problem blocking us from a release, and people scrambling to figure out the problem. We have some software that was returning different results from different environments, and we couldn't figure out the problem. There was a lot of panic in the room, from…

I've been in situations where the whole developer team is trying to solve a production bug, and the managers have never tried to tell us where to focus.

That wouldn't make any sense at all.

Every developer would have their own guesses that they would need to explore and validate, sometimes there's some grouping around where the focus is, but there's usually one guy that's exploring a totally different area to find that bug.

Re: How we spent $30k in Firebase in less than 72 hours

#243

Post by Papirola » Fri, Aug 03, 2018, 2:22 AM UTC

projektir wrote: post #158

blub wrote: post #133

Earlier quoted context omitted.

Software engineering is very hard and can't be taught in bootcamps, blog posts or on the job. It takes years of concentrated effort and quality learning material. A significant number of graduates of computer science couldn't do software engineering after graduation even before. That barrier to entry is designed to protect society from poor quality software and actual software engineers from having to suffer through…

Where can it be taught, then? There are no resources for this right now, mostly people are left to their own devices. Most barriers to entry are not designed to protect anyone, they're designed to preserve power. To protect people from bad products, you need regulation, accreditation, etc.

Universities ?

Re: How we spent $30k in Firebase in less than 72 hours

#244

Post by segmondy » Fri, Aug 03, 2018, 2:33 AM UTC

newscracker wrote: post #88

It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed pos…

Oh you amazing programmer, please share the amazing systems that you are building so we can bow down and learn.

Re: How we spent $30k in Firebase in less than 72 hours

#245

Post by fefb » Fri, Aug 03, 2018, 4:36 AM UTC

catchmeifyoucan wrote: post #233

fefb wrote: post #231

Earlier quoted context omitted.

It won't resolve the problem

AWS Lambda costs about 20 cents per million requests. A billion requests would have put it just a little south of 200-300 dollars. Factor in a few additional read units for DynamoDB, but I don't see it going to total $30,000. The serverless architecture should scale as well.

They are using serverless architecture. Firestore is a "Daas". The latency for loading the page wasn't the real problem, it was the side-effect. The real problem resulted in 40bilion request, they were requesting all the collection of payments in each session. It reaches 16K documents downloaded /session, so they were paying 16K reads documents for session. Firestore gives you 50K reads/day for free, and ask $0.06 for 100K reads. So It was a really bad logical approaching from them. If they did in a best way , like update a document that holds the two accumulators : total money accumulated and total payments created, and read just this document in each session request their bill to show this information for 2M session should be less than $2, because you are reading 1 document for session instead of 16K documents. And pay attention that their system wasn't down, probably the 30s latency was a side-effect of downloading 16K document of each payment information in the client side.

Re: How we spent $30k in Firebase in less than 72 hours

#246

Post by dang » Fri, Aug 03, 2018, 5:18 AM UTC

segmondy wrote: post #244

newscracker wrote: post #88

It seems like they: * didn't have good testing * did no load testing before * had no code reviews done * no design reviews done * had no or very less (useful) application logs * had no change control mechanisms defined or followed (upgrade a framework in production in a matter of minutes or hours as a way to wing it out and pray for it to work out?) * had no or very little automated tests * didn't have a detailed pos…

Oh you amazing programmer, please share the amazing systems that you are building so we can bow down and learn.

Please don't get personally nasty, even if another comment was smug.

Re: How we spent $30k in Firebase in less than 72 hours

#247

Post by projektir » Fri, Aug 03, 2018, 6:01 AM UTC

Papirola wrote: post #243

projektir wrote: post #158

Earlier quoted context omitted.

Where can it be taught, then? There are no resources for this right now, mostly people are left to their own devices. Most barriers to entry are not designed to protect anyone, they're designed to preserve power. To protect people from bad products, you need regulation, accreditation, etc.

Universities ?

Most universities don't even cover something as basic as version control. Monitoring? Forget it.

Re: How we spent $30k in Firebase in less than 72 hours

#248

Post by projektir » Fri, Aug 03, 2018, 6:04 AM UTC

blub wrote: post #181

projektir wrote: post #158

Earlier quoted context omitted.

Where can it be taught, then? There are no resources for this right now, mostly people are left to their own devices. Most barriers to entry are not designed to protect anyone, they're designed to preserve power. To protect people from bad products, you need regulation, accreditation, etc.

There are very good universities focusing on computer science and software engineering in most countries. This is the first step. Then one needs to find a company with a good engineering culture, apply the theory they learned and gather experience. Ideally one should find a qualified engineer as mentor. Self-study and being aware of developments in the profession are the last piece of the puzzle. Yes, some people won…

This seems more like what you would like to happen, not what actually happens to most people, so it doesn't really qualify as resources.

Re: How we spent $30k in Firebase in less than 72 hours

#249

Post by heavenlyblue » Sat, Aug 04, 2018, 8:59 PM UTC

tgsovlerkhgsel wrote: post #73

heavenlyblue wrote: post #8

Earlier quoted context omitted.

Well, a rogue line could potentially as much shut down your app hosted in DO.

That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill. When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down. When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.

Well, then the issue with the Amazon Lambda is not that of it's ability to scale, but that of it's default not to limit your spending.