How we spent $30k in Firebase in less than 72 hours • Hacker News Board

Re: How we spent $30k in Firebase in less than 72 hours

#111

Post by nstj » Thu, Aug 02, 2018, 5:20 AM UTC

boulos wrote: post #93

Disclosure: I work on Google Cloud (but not Firestore or Firebase). For those that didn't read the article, it had a happy ending: > GOOGLE UNDERSTOOD AND POWER US UP! > After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days…

I don’t see why this is being downvoted. The title of the post should be changed to “We were given a $30k credit by GCP after not monitoring our account charges”

Re: How we spent $30k in Firebase in less than 72 hours

#112

Post by abhisuri97 » Thu, Aug 02, 2018, 5:22 AM UTC

ericpauley wrote: post #102

I've been noticing a steady rise of posts from hackernoon by amateur developers who think they'll be the next great tech blogger. I'm not saying I could do any better, but why are these posts suddenly getting so much attention?

As one of those amateur developers who publishes to Hackernoon, it's because they change titles to fit their "How to" type articles + grab attention.

Re: How we spent $30k in Firebase in less than 72 hours

#113

Post by flukus » Thu, Aug 02, 2018, 5:24 AM UTC

RobertRoberts wrote: post #64

Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt? This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them. This reminds me variable rate mortgages. With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I…

> How does anyone justify this risk?

I'm more concerned about the risk mitigation strategies (capping) I'm seeing advocated.

If your servers being pegged you've only got a few customers missing out while it's pegged, maybe even everyone getting service but sub-optimally. You can ride out the wave and everything goes back to normal.

Putting caps in place is like pulling the plug out of the server after the CPU has been at 100% for 5 minutes and not plugging it in until the next billing cycle.

Re: How we spent $30k in Firebase in less than 72 hours

#114

Post by [deleted] » Thu, Aug 02, 2018, 5:29 AM UTC

sitkack wrote: post #40

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

[deleted]

Re: How we spent $30k in Firebase in less than 72 hours

#115

Post by aflam » Thu, Aug 02, 2018, 5:30 AM UTC

megaman22 wrote: post #72

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, a…

Managing those experiences takes effort, otherwise you're sure to end up with the issues you describe. Even when the data makes sense, sometimes people hand you over an unparsable csv with their results, while the conclusions comes from a gut feeling. Sadly not everybody understands standard tools like boxplots, and analyses are often hard to reproduce.

The tooling to make it easy not there yet.

Re: How we spent $30k in Firebase in less than 72 hours

#116

Post by TeMPOraL » Thu, Aug 02, 2018, 5:30 AM UTC

sitkack wrote: post #40

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails.

Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration.

It usually isn't hard to identify which component of your product is misbehaving. Before getting into complex magic of containers and sandboxes and stuff, I'd start with the easiest - looking at the Network tab, at your server's resource use, reading the logs, adding some log statements measuring times in suspected areas, actually profiling the backend code (e.g. with a statistical profiler). This should quickly help you identify where the problem is manifesting itself. Then the search for the cause begins.

Re: How we spent $30k in Firebase in less than 72 hours

#117

Post by notimetorelax » Thu, Aug 02, 2018, 5:33 AM UTC

megaman22 wrote: post #72

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, a…

I think you have a very valid point there, but I would argue it is also important not to react to such situations too strongly. This is something April Wensel talks about here [1] .

[1] bit.ly/2v37AzE

Re: How we spent $30k in Firebase in less than 72 hours

#118

Post by TeMPOraL » Thu, Aug 02, 2018, 5:34 AM UTC

toast0 wrote: post #103

thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

So I completely agree, but let me play devil's advocate. If you do go and track down the problem in your depedency and file a bug, one of two things is likely to happen: they close it and say it's fixed in the latest version or they refuse to accept your bug because it's filed against an old version. Skipping the track it down part and just jumping into upgrading can be a time saver. It works fairly well if you fit i…

> If you do go and track down the problem in your depedency

This is the part the parent's cow-orkers didn't perform. There's nothing wrong in updating a dependency to include the fix for the problem you're experiencing. But the people in question were apparently too lazy/clueless to even track down the problem, opting for randomly upgrading stuff instead.

Re: How we spent $30k in Firebase in less than 72 hours

#119

Post by magnetic » Thu, Aug 02, 2018, 5:36 AM UTC

ShaneOG wrote: post #23

itcmcgrath wrote: post #11

Earlier quoted context omitted.

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.

How did you get your Startup credits?

Re: How we spent $30k in Firebase in less than 72 hours

#120

Post by fefb » Thu, Aug 02, 2018, 5:37 AM UTC

itcmcgrath wrote: post #25

shaunpersad wrote: post #6

I can't say enough good things about Firebase and GCP in general, but I'm always cautious when using Firestore in particular. I usually avoid unbounded queries altogether, and treat it primarily as a key/value store to get by id. When I do use queries, it's always in places where the results have a well-defined limit (usually limit = 1), e.g. finding the most recent X or the highest X. With the above two, you get all…

We've also been rolling out updates to rules to enable you to enforce these types of things. There is performance implications to limit queries for the real-time update system at scale, but for most use cases this shouldn't be a problem. Definitely more we can improve here for control, and we're open to feedback.

Hey, thanks for listening.

I believe in short time would be nice to have a way to use for the query the create, update and write time of a document. Now, I am doing the creat time manage inside of my document with Date.now(), but when I was running a bunch of promises to create the documents, the createTime between the documents that I was manipulating was in same case the same, so my pagination failed.

Another things, like compound queries inside of subcollections should be nice. A way to export all the database for backup.

A flag to alert the Firestore to return the document when I do an update in this document in the same response (one round trip (dynamoDb has it). I know I can reach this goal witht transaction, but I believe it is simpler than a transaction.

A way to update a array without transaction.

Thanks