Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

131–140 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#131

Post by chrisweekly »

dkoston wrote: post #57
cloakandswagger wrote: post #49

Earlier quoted context omitted.

Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.

Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code. Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks. The real story here is that hey didn’t have engineering leadership on the team who knew how to prope…

>"When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code."

^ Best response, hands down.

Re: How we spent $30k in Firebase in less than 72 hours

#132

Post by ocdtrekkie »

boulos wrote: post #93

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days…

Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing.

There's no guarantee if I made the same snafu next week that Google would necessarily be willing to help, but I can absolutely guarantee you that a VM sitting on a Dell PowerEdge I've got lying around would suddenly obligate me to a $35,000 bill, no matter how bad my code.

Ideally, I guess I'd want to see rather than a hard cap, some sort of smart alert that would go "holy crud, this is an unusual spike in the rate of requests" when the delta changed unusually, rather than waiting until I say, hit a high static cost bar or a hard cap that kills the site.

Re: How we spent $30k in Firebase in less than 72 hours

#133

Post by blub »

rspeer wrote: post #110
paulddraper wrote: post #84

Earlier quoted context omitted.

It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.

Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.

Software engineering is very hard and can't be taught in bootcamps, blog posts or on the job. It takes years of concentrated effort and quality learning material.

A significant number of graduates of computer science couldn't do software engineering after graduation even before.

That barrier to entry is designed to protect society from poor quality software and actual software engineers from having to suffer through picking up the broken pieces after those people that were helped to jump the barrier.

Re: How we spent $30k in Firebase in less than 72 hours

#134

Post by cube2222 »

onion2k wrote: post #126
r3vrse wrote: post #76

I spend a fair amount of time on HN. Among many, I think this article is probably the most succinct endictment of ADHD-ridden "modern" web programming/ecosystem practices I've read. It's so sad to me that while the name dropping and churn for frameworks and languages continues, frenzied and unabated -- basic (pun sort of intended) analysis and problem-solving techniques go out the proverbial window. Why learn to thin…

Critical thinking is a learned skill. For most developers it comes with experience, and many developers in startups are often yet to learn it. Thinking well under pressure is even harder. This is more a problem about startups using inexperienced developers than anything related to what they're building or which tech they're using.

Indeed, if your building anything, good metrics are one of the most important things. Didn't see them having any metrics apart of Google analytics.

Re: How we spent $30k in Firebase in less than 72 hours

#135

Post by planetjones »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Yep with performance like that it’s highly likely going to be a problem involving the persistent store of the application. The first instinct should have been to enable some tracing or logging to see where the time was being spent.

Re: How we spent $30k in Firebase in less than 72 hours

#136

Post by flukus »

ericpauley wrote: post #102

I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?

> but why are these posts suddenly getting so much attention?

The same reason we slow down for car crashes, morbid curiosity. I don't think there is anything "sudden" about it though, we even have sites like thedailywtf dedicated to this level of idiocy.

Re: How we spent $30k in Firebase in less than 72 hours

#137

Post by boulos »

ocdtrekkie wrote: post #132
boulos wrote: post #93

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days…

Perhaps somewhat optimistically, I assume most of the commenters read the whole article, but I don't think the happy ending abates the concern. It's great the Google Cloud team was able to bail them out afterwards, but the fact that they were able to rack up a $35,000 cost on a code mistake still highlights one of the major flaws with pay-as-you-go cloud computing. There's no guarantee if I made the same snafu next w…

Right, I don’t deny that (and didn’t mean to imply this isn’t a problem). At the time of my comment it did seem like most people stopped before the end :).

Budgets and quotas are really tricky as Dan pointed out elsewhere in this thread. App Engine has had default daily budgets (that you can change) forever, but then you run into people saying “What the hell, why did you take down my site?!”.

In this case, they even intentionally pressed forward once they saw their bill was going up. If this had been say a static VM running MySQL with a “SELECT *” for every page view, the site would likely have just been effectively down. For some customers, that’s the wrong choice, even in the face of a crazy performance bug.

That said, we (all) demonstrably need to do better at defaults as well as education (the monitoring exists!).

Re: How we spent $30k in Firebase in less than 72 hours

#138

Post by michalprzy »

megaman22 wrote: post #72
sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, a…

Unfortunately I often see developers asking for complex diagnose in urgent emergencies. When it's literally like a 30 minutes reaction time frame. So from business pov there should be first quick and dirty unperfect analysis which is necessary for emergency decisions. Sometimes you can just prolong caching time as quick fix, but sometimes there is no such option. Programmers like nice and calm working environment but business often is very different place. So, please have in mind what business goals are. And yes, I see there is a lot of stupid managers in It and basically you're right, we should use precise data. Just one point: when available...

Re: How we spent $30k in Firebase in less than 72 hours

#139

Post by bushin »

blub wrote: post #133
rspeer wrote: post #110

Earlier quoted context omitted.

Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.

Software engineering is very hard and can't be taught in bootcamps, blog posts or on the job. It takes years of concentrated effort and quality learning material. A significant number of graduates of computer science couldn't do software engineering after graduation even before. That barrier to entry is designed to protect society from poor quality software and actual software engineers from having to suffer through…

Type with one hand, make your software better!
Post reply on HN